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ABSTRACT 


Systems engineering (SE) competencies are defined based on the knowledge, skills, and 
abilities (KSAs) necessary for a systems engineer to perform tasks related to the 
discipline. Proficient systems engineers are expected to be able to integrate, apply, and be 
assessed on these KSAs as they develop competencies through their education and 
training, professional development, and on-the-job experience. The research conducted 
by the Naval Postgraduate School, assessed where SE graduate students stood as far as 
developing the necessary competency levels they need to be successful systems 
engineers. A survey methodology was used to achieve this objective. Systems 
engineering students enrolled in SE courses at the Naval Postgraduate School represented 
the population surveyed. 

Survey items were written with the intent to capture self-efficacy for knowledge 
and skill sets as a subset of the overall set of competencies required for systems 
engineering, namely critical thinking, systems engineering, teamwork and project 
management. A total of four surveys were administered to two SE cohorts. Results show 
that self-efficacy in systems engineering can be reasonably assumed to be positively 
affected by a graduate level educational program. The implications of the research can be 
used to develop structured curriculum content, assessment, and continuous process 
improvement techniques related to the development of SE learning, and to develop more 
valid and reliable instruments for assessing what systems engineers need to learn, need to 
know, and need to do. 
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EXECUTIVE SUMMARY 


This research focused on how self-efficacy might be used as a means to assess the 
competency development of systems engineering (SE) students. The method was also 
developed as a first step in attempting to predict future work performance of systems 
engineers. Self-efficacy, as it relates to research within systems engineering, is defined as 
an individual’s belief that he or she can be proficient in skills necessary to become a 
systems engineering professional. SE self-efficacy is also defined as one’s beliefs in 
his/her ability to learn SE knowledge, skills and abilities (KSAs). The research is 
significant as current studies suggest that proficient systems engineers are expected to be 
able to integrate and apply competencies throughout their education, professional 
development and experience. The thesis presents a brief introduction to the Naval 
Postgraduate School’s (NFS’) competency model, how the competency model relates to 
the SE Competency Survey, an in-depth analysis of the related self-efficacy research, 
methodology, and the results of the SE Competency Survey instrument. 

The study determined how self-efficacy items, identified in the SE Competency 
Survey instrument, relate to a SE competency model developed at NFS. Systems 
engineering competencies were identified as the educational learning objectives needed 
to be acquired by systems engineers to perform tasks related to the discipline. The KSAs 
were grouped into cognitive and affective domains. As such, the SE Competency Survey 
presents the cognitive and affective KSAs as measures of performance. The SE 
Competency Survey instrument is based on a subset of the SE learning domains that 
systems engineering students are expected to be proficient in (systems engineering, 
critical thinking, project management and teamwork). University faculty and engineering 
organizations can use the instrument to assess areas of skill sets that students and/or 
entry-level engineers are strong, weak in, or need to develop further. 

A literature review led the team to believe that a gap exists between being able to 
assess SE KSA and measure the competency of students/entry level engineers as a way to 
predict future career performance. The research presents an assessment of the roles self- 
efficacy and cognitive/affective skills play in indicating future performance in systems 
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engineering related careers. The role of self-efficacy in indicating performance in SE 
related careers can be seen as that of a psychosocial construct. It is influenced by one’s 
own sense of capability and by the education, training and support received by others. 
The NFS research team cited findings by the National Security Leadership Foundation, 
which suggest that an individual’s level of self-efficacy is a motivational predictor of 
how well a person will perform tasks and can be utilized and developed to leverage 
performance enhancing advantages; an individual’s perceived self-efficacy greatly affects 
his/her ability to achieve domain specific tasks. Self-efficacy serves as a regulator across 
a broad range of functioning, as the key determinant for cognitive and affective 
processes. Similarly, the research is partially based on Prawat’s finding that both 
cognitive and affective skills are determined to be vital in indicating performance in SE 
because having a broad range of KSAs enables knowledge acquisition and application 
(1989). The NPS team considers that acquiring domain specific knowledge has the 
advantage of allowing one to facilitate the application, or performance, of work. 

The study of self-efficacy is presented with a detailed explanation of the statistical 
tests, factor analysis (FA) and Cronbach’s alpha, which were used to assess the reliability 
of the survey instrument. The FA process and the determination of Cronbach’s alpha 
added weight to the reliability of the SE Competency Survey as a tool to assess the 
perceived self-efficacy of systems engineering students to perform tasks within the SE 
domains. Current research findings suggest that using questionnaires, such as the SE 
Competency Survey, can aid in the identification of complex, latent psychological traits— 
in this case, self-efficacy. Current research also suggests that participants in a study such 
as this, are not likely to perceive themselves to be poor in a competency area they think is 
very important (Moore and Rudd 2005, 76). With this in mind, and to add weight to the 
findings, our analysis shows that Cronbach’s alpha was determined to be high— 
providing support that the NPS survey data results have a relatively high internal 
consistency for evaluation. 

To provide verification of the research method used to measure acquisition of 
domain specific knowledge, the NPS team used two pre- and post-survey renditions of 
the SE Competency Survey. The goal of the survey was to gauge the perceived levels of 



self-efficacy among systems engineering students, over time, in four systems engineering 
domains as they continued through the systems engineering program at NPS. Survey 
items were written with the intent to capture domain specific skills and knowledge sets 
within critical thinking, systems engineering, teamwork and project management. The 
discussion provided describes detailed findings on the results of the survey research on 
each of the four domains, as well as commentary on possible reasons for the findings. 

In general, the teamwork results indicate mixed findings as the results for the first 
cohort did not show significant differences in the average response rates towards many of 
the teamwork aspects, while the second cohort seems to have improved at significant 
levels. Among the first cohort’s results, the skill was observed to be significant at p<.01, 
with a percent increase in pre- and post-results of nine percent. The second cohort’s 
percent increase between pre- and post-average scores was 30 percent, and was 
statistically significant at p<.001. A possible explanation for this observation is that lead 
faculty incorporated course work that addressed teamwork into the curriculum of the 
second cohort (RT19A), but not the first cohort. This was because, for the first cohort, it 
was assumed the students could effectively work in teams. This structural method may 
have impacted the results. This finding is indeed interesting as research has found 
teamwork in engineering has the advantage of solving complex problems (Ponton et al. 
2001, 249). 

The critical thinking data analyses of the first cohort’s results seem to indicate 
that the self-efficacy scores in critical thinking did not go up enough to exceed rejection 
criteria for traditional hypothesis testing—which would contend that there would be a 
significant difference between pre- and post-averages. All but two of the skills surveyed 
demonstrated higher averages in the post-survey results—meaning that even though the 
gains were not significantly different, there was at least a general sense of improvement 
among students. An analysis of the second cohort yields markedly different results, with 
most of the post-mean differences relating to critical thinking self-efficacy skills 
statistically significant from pre-survey results at p<.001. In reviewing the critical 
thinking results, it should be noted that the structural format in teaching the first cohort 
involved referring the students to textbooks on the topic of engineering reasoning, 
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whereas the second cohort was more formally taught from lectures (i.e., through parts of 
course work and seminars) on what critical thinking is and how it is used. The NPS 
research team feels that this change in structure may have impacted the survey results. 

For systems engineering, the first cohort’s results showed significant differences 
in several areas. Two skills in particular showed significant differences between pre- and 
post-results at p<.01. These were the student’s ability to use projected warfighting 
capability gaps to estimate the future need for a new system and to describe what 
resources will be necessary to conduct tests to demonstrate that a system meets its 
requirements. One self-efficacy item in particular reflected statistically significant pre- 
and post-survey results at p<.001, which was the student’s ability to pick a technology for 
a system considering that the technology readiness level is estimated, and uncertain. 
There were significant differences in means observed in all self-efficacy items relating to 
systems engineering. These results may be due to the change in structural format of 
instruction delivery between that of the first and second cohort. The first cohort 
integrated their authentic project within each of the courses in the curriculum. The second 
cohort had three additional consecutive project courses added to their curriculum in 
which the instructor reviewed all SE methods and processes in the context of the project. 
This facilitated near transfer of the learning of SE methods and processes to the authentic 
project context. The NPS team feels that this approach to teaching systems engineering 
concepts in context of authentic-learning projects seems to improve self-efficacy and is 
more effective in teaching. 

Eor project management, most of the first cohort’s results did not go up enough to 
exceed traditional rejection criteria for hypothesis testing. An exception to this 
observation was in the skill of estimating quantitatively the likelihood and relative impact 
of events on a development project that represent the most pressing risks, which had an 
increase of 11 percent, and the difference in mean scores was statistically significant at 
p<.05. Results of the second cohort’s survey items relating to project management 
showed statistically significant differences in the mean scores at p<.001—exceeding 
rejection criteria for traditional hypothesis testing. Similar to the analysis discussed on 
systems engineering, the project management results may be due to the change in 
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structural format of instruction delivery between that of the first and second cohort. As in 
the case in the systems engineering domain, the second cohort was explicitly taught how 
SE skills relate to the project via learning activities. The NFS team feels that assisting 
them in near transfer of knowledge and ability to their project may have influenced the 
survey results to reflect an increase in self-efficacy. 

Results from the survey analysis led to the conclusion that self-efficacy in systems 
engineering can be reasonably assumed to be positively affected by a rigorous 
educational program and training. As such, the NFS team feels that the SE Competency 
Survey can be used by university faculty and engineering enterprises to measure self- 
efficacy in systems engineers in an overall effort to predict future career performance. 
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I. 


INTRODUCTION 


Recent studies into career development models have led to the general finding 
that contemporary firms are searching for well-rounded individuals, who possess both 
technical and non-technical skills (Joshi et al. 2010, 3). Scholarly studies examining 
hiring practice trends through content analysis of job advertisements and job types, have 
observed that stakeholders prefer to employ well-rounded employees with business 
knowledge, interpersonal skills and technical skills (Joshi et al. 2010, 3). According to 
Joshi et ah, “Students gain their awareness of the importance of technology related job 
skills from the academics who teach them” (2010, 3). As such, a goal of university 
curricula and systems engineering enterprises should be to educate to provide the 
necessary technical and non-technical background to enable students to become 
proficient in performing systems engineering competencies. Systems engineering (SE) 
competencies are defined as the knowledge, skills and abilities (KSAs) necessary to 
perform tasks related to the discipline. Proficient systems engineers are expected to be 
able to integrate and apply such competencies throughout their education, professional 
development and experience. 

Pertinent to our goal as an academic institution to educate students towards 
becoming proficient engineers was the need to identify the current KSAs major 
stakeholders seek in the employment of systems engineers. When considering the 
development of key technical and non-technical KSAs, we also examined the role of self- 
efficacy as a regulator of cognitive and affective learning processes, as studies have 
shown perceived self-efficacy contributes to cognitive and affective development and 
functioning (Bandura 1993, 1). 

This thesis describes the process that was undertaken by the Systems Engineering 
Department at the Naval Postgraduate School (in Monterey, California), to measure and 
evaluate the level of competency, or “self-efficacy,” perceived by systems engineering 
students. To obtain an understanding of where graduate systems engineering students 
stand as far as developing the necessary competencies they need to be successful systems 

engineers, we developed a series of surveys to gauge their level of self-efficacy. 
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Systems engineering self-efficacy, in this study, is defined as the individual’s 
belief that he or she can be proficient in skills necessary to become a systems engineering 
professional. System’s engineering self-efficacy also refers to one’s beliefs in his/her 
ability to learn systems engineering skills. 

Self-efficacy can be further defined as a “psychological characteristic known for 
its ability to have an enduring influence on related behavior and interests” (Mau 2003, 
234). Expanding on this description, self-efficacy has also been described as a formative 
social characteristic which can have a spiral effect, in that once it is well established in an 
individual, it can influence that individual to make choices and attempt tasks that further 
strengthen and increase confidence in his or her abilities to perform tasks within a domain 
(Lucas and Cooper 2004, 12). An objective of this study was to determine whether self- 
efficacy, alongside being a psychological trait, can be taught, or enhanced, through 
education. Research suggests that when looking at self-efficacy, particularly in 
engineering students, it is important for an engineering professor to understand how self- 
efficacy relates to learning concepts. 

The context of how engineering students learn is often categorized using Bloom’s 
taxonomy. Engineering education involves learning objectives, which are typically 
organized around Bloom’s taxonomy of cognitive and affective processes. Cognitive and 
affective processes within Bloom’s taxonomy refer to levels of observable actions which 
indicate learning is occurring. The cognitive domain involves knowledge and the 
development of intellectual skills, whereas the affective domain deals with the 
motivations and attitudes involved in learning (Clark 1999). 

In keeping engineering learning objectives in mind, it takes a great amount of 
effort on behalf of the professor to use mastery experiences, vicarious experiences, verbal 
persuasion, and physiological/emotive arousal strategies within instruction, to improve 
self-efficacy (Ponton et al. 2001, 249). As such, we hope findings from this study can be 
used to gauge self-efficacy in systems engineering students and to implement changes to 
improve systems engineering instructional delivery strategies. 
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The thesis begins with a discussion on the methodology that was utilized to 
acquire and analyze data necessary to evaluate the level of self-efficacy of systems 
engineering students to independently perform systems engineering tasks. The review 
includes a discussion on the survey methods used to study self-efficacy in systems 
engineering, which included the process of utilizing the statistical method of rotated 
factor analysis as a means to test for the validity, or effectiveness, of the survey 
instrument to capture reliable data. 

The goal of the survey was to gauge the perceived levels of self-efficacy among 
systems engineering students, over time, in four systems engineering domains as they 
continued through the systems engineering program at NFS. Survey items were written 
with the intent to capture domain specific skills and knowledge sets within critical 
thinking, systems engineering, teamwork and project management. The discussion 
provided describes detailed findings on the results of the survey research on each of the 
four domains, as well as commentary on possible reasons for the findings. Results from 
the survey analysis led to the conclusion that self-efficacy in systems engineering can be 
reasonably assumed to be positively affected by a rigorous educational program and 
training. 

A. RESEARCH QUESTIONS 

The background of this topic has presented several issues that were considered in 
the formulation of the research questions. The following research questions have been 
developed in response to the previously outlined research objectives. These research 
questions will guide the path of this research that will culminate in their answers and 
conclusions provided in Chapters IV and V. 

1. Primary Research Question 

The primary research question will help determine whether self-efficacy can be 
reliably measured to validate the attainment of domain specific skills among systems 
engineering students. 

• Primary research question 1: Can self-efficacy be used as an indicator of 
systems engineering competency level achievement? 
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2. Secondary Research Questions 

While examining the possible methods to reliably measure and assess self- 
efficacy as a competency construct, a set of secondary research questions developed. 

• Secondary research question 1: What is self-efficacy in systems 
engineering? 

• Secondary research question 2: How might self-efficacy be used in 
systems engineering? 

• Secondary research question 3: What methods can be used to assess self- 
efficacy? What methods can be used in systems engineering? 

• Secondary research question 4: Can a survey instrument be used to 
identify achievement of specific knowledge, skills and abilities (KSAs) as 
indicators of competency in performing systems engineering tasks? 

• Secondary research question 5: If possible within time, participant, and 
resource constraints, can the systems engineering survey instrument be 
used by other engineering disciplines as a model to implement the self- 
efficacy based method, and can it be tailored to meet an organization’s 
requirements with minimal effort/changes? 

• Secondary research question 6: What roles do self-efficacy and 
cognitive/affective skills play in indicating performance in systems 
engineering related careers? 

B. PURPOSE/BENEFIT 

The benefit of the study will be two-fold. First, a direct result of the research will 
be a survey instrument, which can be used to reliably measure self-efficacy as it relates to 
competency levels in systems engineering. The research will serve to test and verify the 
usefulness of the self-efficacy survey instrument in determining levels of competency 
achievement. Second, the survey instrument will be a tool that naval system commands 
and educational institutions can use to measure and verify the level of competency their 
employees/students have achieved. This can then be used by organizations to track 
workforce development, to confidently assign work tasks, to assess the ability to perform 
standard systems engineering tasks, and to create individual development plans for career 
management. 
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C. THESIS STATEMENT 

Our research maps competencies as theory based indicators of self-efficacy to 
establish, with confidence, which measures of graduate level education would be 
expected to indicate higher-levels of SE self-efficacy. 

D. RESEARCH METHODS 

The research methods discussed in the following chapters will discuss and 
describe the process, methods and products developed by NFS to answer the 
aforementioned research questions. The primary NFS research product was a survey 
instrument, which was designed to gather information from students on self-efficacy. 
Specifically, the survey was given over a period of two years to two different cohorts. 
The research methods used to examine the reliability of the survey instrument to measure 
self-efficacy as a latent psychological construct included factor analysis and Cronbach’s 
alpha. 

E. CHAPTER SUMMARY 

In this chapter an introduction to the research topic was presented along with a 
stakeholder analysis on the needs of industry, which concluded that there is a need for 
engineers who are competent in both technical and non-technical skills. The introduction 
followed with a description of self-efficacy, and how it relates to systems engineering. 
The first chapter concluded with a brief introduction to the survey instrument and 
methodology used to conduct the research in an effort to answer the research questions, 
which were also provided. Finally, the purpose and benefit of the research was stipulated 
to be two-fold. The research led to the development of a survey instrument, which can be 
used to reliably correlate self-efficacy to competency levels in systems engineering. 
Alongside the SE competency career development model, the survey instrument was 
suggested as a resource whereby organizations can: track workforce development, assign 
work tasks, assess the ability of engineers to perform standard SE tasks, and create 
individual development plans for career management. 
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II. LITERATURE REVIEW 


The following is a brief literature review of past and current research on self- 
efficacy constructs, and in particular addresses literature that supports the importance of 
self-efficacy and its influence on academic achievement and work performance. The 
review suggests relevant connections can be made between self-efficacy and social 
cognitive theory (SCT), entrepreneurship (Lucas et al. 2009), development of leadership 
and technical skills (Joshi et al. 2010), and meta-cognition (Lang 2012). Metacognition, 
the knowledge of one’s own cognition and the regulation of such skills, has been found to 
be an important factor in successful task performance (Kraiger, Ford and Salas 1993). 

Several studies extending self-efficacy theory have related it to vocational 
behavior, suggesting expectations of self-efficacy aid in the determination of career 
decisions, can serve as predictors of achievements of men and women, and may be 
particularly useful in understanding women’s career development (Hackett and Betz 
1981). However, the focus of our discussion is a literature review of the lessons that can 
be gleaned from research on the importance of self-efficacy in indicating a predilection 
towards successful work performance. 

A. SELF-EFFICACY IN RELATIONSHIP TO SOCIAL COGNITIVE 

THEORY 

According to Lent et al., “In a review of Social Cognitive Theory (SCT) and 
career and academic interests. Lent et al. (1994) provide evidence that self-efficacy in a 
particular domain helps develop the individual’s initial interest in a corresponding career 
area, followed by the selection of career paths, and then supports the higher levels of both 
academic performance and persistence in those domains” (2009, 740). We argue that 
such “higher levels” of academic performance are indicative of the development of 
cognitive and affective processes, which we have identified via the NPS competency 
model, as discussed in Chapter III, on the methodology and results of the NPS research. 

SCT posits that human agency (defined here as intentional behavior) is affected 
by the “reciprocal interaction between personal behavior, internal personal factors (i.e.. 
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one’s cognitive, biological, and affective characteristics) and the environment (i.e., 
everything external to the individual) where all three determinants interact with various 
magnitudes of influence that is context dependent” (Ponton et al. 2001, 247). 

Foundational research performed by Lucas et al. (2009), summarizes the role self- 
efficacy has in enabling students to pursue careers involving innovation. Lucas et al. 
suggest, “based on Social Cognitive Theory, one would expect that individuals with a 
high self-efficacy in a particular domain both perform work tasks better, and are more 
innovative in the way in which they go about their work in that domain” (2009, 740). 
They suggest that self-efficacy is central to career choice and in accepting careers in more 
challenging fields, particularly in science and engineering. Furthermore, Lucas et al. cite 
Bandura’s findings that suggest that domain self-efficacy (in this context, the confidence 
individuals have in their ability to undertake a range of activities within a specific 
domain) applies to career selection. 

1. Work Performance 

In referencing Stajkovic and Luthans (1998), Lucas et al. contend that the role of 
self-efficacy continues to be important once individuals have entered their careers, 
predicting better work performance (Lucas et al. 2009, 740). In addition, they go on to 
explain, “Causal inferences about the effect of self-efficacy on work performance are 
supported by experimental research that uses random group assignment and the 
manipulation of self-efficacy” (Lucas et al. 2009, 740). Lucas et al. (2010, 738) have 
explored the correlation between various workplace experiences and venturing and 
technology self-efficacy. They also focus heavily on the development of self-efficacy 
through authentic work experience. 

For work experience to increase self-efficacy for future entrepreneurial and 
technical applications, it must be authentic (Lucas et al. 2009, 741). Lucas et al. define 
authentic as being seen by the students as having required similar skills and levels of 
performance that they believe will be expected of them if they pursue innovative careers 
in engineering (2009, 741). They also contend that if self-efficacious individuals set 
themselves more challenging goals and are prepared to take on unfamiliar tasks then, as 
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previously mentioned, self-efficacy is expected to greatly influence the likelihood that the 
individuals will attempt innovative behavior (Lucas et al. 2009, 740). Their research 
measures two types of self-efficacy, one for venturing skills (starting one’s own business; 
entrepreneurial) and the other for technology. They then map the factors in social 
cognitive theory that predict higher self-efficacy to the elements of undergraduate work 
experience. 

Lucas et al. (2009) achieved the aforementioned objectives via a survey 
methodology, which we aimed to emulate in our study. Their model and research 
examined the effects of IT skills self-efficacy and IT importance on student’s attraction to 
IT careers. A key difference between our research emphases is that Lucas et al. discuss 
and describe implications of self-efficacy and background characteristics of survey 
participants, types of industry placement and the university the students are drawn from, 
in a test for possible sampling bias. We did not ask for student demographic and 
background data in an effort to encourage honest responses, and as such, do not have a 
valid means to match student demographic and background data with responses for any 
correlation analysis. 

The Lucas et al. (2009) self-efficacy survey items intended to measure venturing 
and technical self-efficacy in undergraduate engineering students who had previous work 
placements. Their survey also included questions that characterized different elements of 
work experiences, including the orientation of the company at which the students worked, 
the nature and difficulty of assigned work, and the presence of role models where the 
students were placed (Lucas et al. 2009, 739). Of particular interest to our study was the 
survey scale, which asked students to rate their self-confidence in performing science and 
technology related tasks, as a measure of engineering competency. The survey instrument 
scale ranged from 0 (not at all confident) to 10 (completely confident). Items were 
written to measure self-confidence in performing tasks undergraduates would be expected 
to perform if they were to be proficient in technology related work. These skills and 
abilities included understanding the limits of technology, translating user requirements 
into technical requirements, and leading a team of technical professionals. 
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Lucas et al. (2009) then used regression analysis to examine correlations among 
student background, organizational and other predictor variables. The results of their 
survey data analysis led to the conclusion that on average, work experience had little 
effect on student self-efficacy. Yet, when the survey data reflected that the student 
experiences had the qualities generally accepted to predict enhanced self-efficacy, strong 
differences were observed. Lucas et al. suggest that these generally accepted constructs 
are: authentic experience, social influence, vicarious experience and emotional states 
(2009). Their results indicate that authenticity, feedback on performance, and how well 
students feel they have performed, are the dominant predictors of self-efficacy (2009). 

Lucas et al. explain, “The effect of workplace experience, generally found in 
literature to be important, is widely viewed by university instructors as having a 
consequential impact on student readiness for the world of work” (2009, 739). Lucas et 
al. stipulate that one approach to prepare university students for innovative technical 
careers is to emphasize methods of education that replicate aspects of real work 
experience (2009, 739). They also very astutely point out that a “major task for higher 
education is the development of employees who not only have the right skills and 
attitudes, but also the ability to learn from experience and adapt within a dynamic and 
rapidly changing environment” (Lucas et al. 2009, 739). The role of research should be to 
determine what aspects of a student’s experience enhance self-efficacy for venturing and, 
independently, self-efficacy for technology (Lucas et al. 2009, 739). Lucas et al. state, 
“Pinquart et al. (2003) suggest that increased knowledge and understanding of the 
industry environment as a workplace helps to ease the transition from education to 
employment” (2009, 740). We argue that the use of our survey instrument can aid in the 
identification of where students are at in gaining the required knowledge and 
understanding of industry practices to be proficient engineers. The NFS’s competency 
development model can help to identify at which career developmental stage students 
should gain the knowledge, which would, we assert, ease the transition from their 
education to subsequent work assignments. Research by Lucas et al. (2009) does not 
answer whether or not this requires a higher level of cognition, our research fills this gap. 
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Furthermore, we contend that our research aids in the identification of the characteristic 
experiences and knowledge students need to strengthen self-efficacy in technical and 
non-technical fields. 

B. SELF- EFFICACY: PREDICTOR OF WORK PERFORMANCE 

Stajkovic and Luthans (1998) surveyed 114 research reports on self-efficacy and 
work-related performance and found that it consistently predicts higher levels of work 
performance across various types of organizations and activities. They researched the 
correlation between self-efficacy and work-related performance, hypothesizing that there 
is an overall positive relationship between self-efficacy and work-related performance 
(Stajkovic and Luthans 1998, 241). A meta-analysis (empirical research examining the 
results of all 114 studies) performed by the team aggregated and assessed individual 
research findings pertaining to the relationship between self-efficacy and work-related 
performance (Stajkovic and Luthans 1998, 240). In their meta-analysis, they investigate 
the overall magnitude of the relationship between the two constructs (Stajkovic and 
Luthans 1998, 240). 

Their findings indicate that a variable of potential moderating importance 
regarding the relationship between self-efficacy and work-related performance is task 
complexity, and that past research on self-efficacy has shown it is related to several 
work-performance measures. Such measures are “adaptability to advanced technology, 
managerial idea generating/performance, skill acquisition, newcomer adjustment to an 
organizational setting and naval performance at sea” (Stajkovic and Luthans 1998, 240). 

I. Self-Efficacy: Predictor of Academic Performance and Perceived 
Career Options 

Lent et al. (1984) tested their hypothesis that efficacy expectations should relate 
to persistence and success in pursuing educational and career goals. Their findings 
suggested that students’ beliefs about their ability to complete the educational 
requirements of various engineering and science fields were indicative of subsequent 
academic performance. For instance, students reporting comparatively strong self- 
efficacy were generally found to achieve higher grades and were much more likely to 
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persist in technical/scientific majors over a one-year period compared to those with low 
self-efficacy. Subsequent research by Lent et al. (1986) served to extend results of other 
studies (Hackett and Betz 1981) by assessing the relation of efficacy beliefs to perceived 
career options in science and engineering students. In particular, they studied the 
applicability of self-efficacy, alongside “ability, achievement, and interest measures in 
predicting academic success, persistence, and range of perceived career options in 
technical/scientific fields” (Lent et al. 1986, 265). 

Self-efficacy was assessed via survey instruments that measured self-efficacy, 
career indecision, self-esteem, expressed vocational interests, and range of perceived 
vocational options in technical/scientific fields (Lent et al. 1986). Similar to our mode of 
survey delivery, their surveys were administered as pre and post surveys, specifically 
during the first and final class sessions. Self-efficacy was assessed by asking subjects to 
indicate whether they believed they could successfully perform the educational 
requirements (ER) and expected job duties in 15 science and engineering fields. 
Individual levels of self-efficacy (ER-L) scores were obtained by summing the number of 
fields subjects believed they could complete. Strength of self-efficacy (ER-S) was 
assessed by having subjects estimate their degree of confidence in their ability to 
complete the ER and job duties. Similar to our study, self-efficacy to perform a task was 
indicated using a numerical scale. Whereas our scale was 0 to 100, with 0 representing no 
confidence and 100 representing complete confidence, Eent et al. used a range of one to 
ten, from completely unsure (1) to completely sure (10) (Eent et al. 1986, 266). Then, 
strength scores for each subject were calculated by dividing the summed strength 
estimates by 15, the total number of major/career fields (Eent et al. 1986, 266). It is 
noteworthy to emphasize here that again, similar to our study, students were asked to rate 
their ability to perform specific accomplishments considered to be critical to academic 
success and performance in science and engineering fields. 

Subsequent findings by Eent et al. concluded that student’s beliefs about their 
ability to complete the educational requirements of various science and engineering fields 
were predictive of subsequent academic performance (1984, 265). Eurthermore, 
hierarchical regression analyses indicated that “self-efficacy contributed signification 
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unique variance to the prediction of grades, persistence, and range of perceived career 
options in technical/scientific fields” (Lent et al. 1984, 265). 

Thus, self-efficacy has been shown to be a useful measure in the prediction of 
academic performance, yet there is a lack of research to support the assertion on the 
usefulness of self-efficacy as a reliable indicator in relation to competencies that define 
career performance. Our research aims to close this gap. 

C. RESEARCH ON SELF-EFFICACY IN INFORMATION 

TECHNOLOGY—A PARALLEL TO SYSTEMS ENGINEERING 

Similar to SE self-efficacy, information technology (IT) self-efficacy refers to 
one’s beliefs about his/her ability to learn IT skills necessary to conduct IT work (Joshi et 
al. 2010, 2). High self-efficacy for a domain such as IT would be theorized to lead to 
choice (versus avoidance) of IT coursework and college majors, to facilitate performance 
on IT related tasks, and to lead to persistence during difficult academic/career related 
experiences (Joshi et al. 2010, 2). As Joshi et al. explain, “Students gain their awareness 
of the importance of IT job skills from referent others—including the academics who 
teach them, peers who socialize with them, and parents who guide them” (2010, 3). This 
sense of IT skills importance refers to one’s beliefs about the value of IT skills necessary 
to be successful in the IT profession (Joshi et al. 2010, 2). Our research does not correlate 
a student’s perceived level of importance of SE skills—perhaps this could be a future 
area of research. 

Similar to Eucas et al., Joshi et al. analyzed self-efficacy survey responses of 
university students with the intent to study gender differences (2010, 2). Joshi et al. 
(2010) conducted a three-part study that aimed to enhance the overall understanding of 
attitudes about the IT field and subsequent interest in an IT career. The significance of 
their research, as it relates to the NPS study on SE self-efficacy, is that both research 
teams are aiming to study the dynamics between self-efficacy in technical fields. So, for 
the purposes of NFS’s research, their findings are particularly relevant and noteworthy. 

Phase I of the Joshi et al. project involved an analysis of three categories of IT job 
skills (technical, human and business), across three genres of texts: scholarly articles. 
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practitioner literature and online job ads (2010, 2). Likewise, in a study of employment 
opportunities for recent college graduates, Wong, von Hellens and Orr used both IT job 
ads, surveys and interviews with stakeholders and determined that non-technical skills are 
increasingly more important than technical skills (Joshi et al. 2010, 3). The Phase II study 
revealed that while masculine stereotypes are applied to most technical skills, and 
feminine stereotypes to most interpersonal skills, gender neutral skills (i.e., initiative, 
ability to work under pressure, critical thinking, and problem solving) have emerged as 
important constructs today’s students must possess. Phase III of their study focused on 
the development, validation and effects of IT related self-efficacy and IT skills 
importance on IT career intentions (Joshi et al. 2010, 2). To achieve this objective, Joshi 
et al. used a survey methodology similar to Lucas et al. and NPS research. The Joshi et al. 
survey asked students to specify their level of self-confidence in their perceived ability to 
“learn and engage” in 36 IT skill items. Results were subjected to factor analysis, to 
uncover the underlying factor structure of the IT self-efficacy construct. Two constructs 
emerged from the factor analysis results: non-technical and technical self-efficacy. The 
significance of the research of Joshi et al., particularly within Phase III, is that it is 
another example of a successful attempt at measuring a latent psychological trait using a 
survey model. It adds weight to the idea that task specific measures in the form of a 
survey, might be superior to other approaches to self-efficacy assessment (i.e., ER) 
(Hackett and Betz 1984). 

1. Stakeholder Analysis 

According to Joshi et al., “The process of consulting with stakeholders to solicit 
feedback regarding the importance of skills and the quality of training that universities 
provide, ensures that education meets the needs of industry” (2010, 2). Stakeholder 
analyses assessing the need for technically competent professionals in industry have led 
to the observation of the shifts in the composition of the IT workforce and accompanying 
changes in the relative importance of IT skills (Joshi et al. 2010, 3). For instance: 

low skilled IT job positions (such as data entry and computer operators) 
are disappearing. IT jobs are shifting from traditional, centralized MIS 
[management information systems] divisions out to business units. 
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necessitating employees to have more business knowledge and stronger 

interpersonal/management skills for working with clients. (Joshi et al. 

2010 , 3 ) 

Joshi et al. note, “In addition to studies that examine the perceptions of 
stakeholder groups, scholars have investigated shifts in IT skills demand and job types 
through content analysis and job advertisements” (2010, 3). Findings have suggested that 
systems analysts mostly value their non-technical roles and skills, whereas systems users 
allocate more significance to the analyst’s technical skills (Joshi et al. 2010, 3). In a 
similar study of webmasters and users, it was found that users valued the importance of 
technical skills, whereas webmasters asserted organizational skills were more important 
to their successful job performance (Joshi et al. 2010, 3). 

McGee suggests that the lack of non-technical skills will result in a failure of an 
IT professional to be promoted to a manager position (Joshi et al. 2010, 3). Stakeholder 
analyses also indicate that “the high failure rate of IT projects is not attributed to the 
technological complexities of projects or the technical incompetency of the project teams. 
Rather, it is attributed to the non-technical skills within the IT project management 
teams” (Joshi et al. 2010, 3). 

D. ENHANCING SELF EFFICACY AND COGNITIVE MOTIVATION 

Literature suggests that the presence of self-efficacy helps to motivate students to 
delve into engineering course work, which aids in their development as a capable 
professional (Ponton et al. 2001, 249). Sources of self-efficacy evaluations are mastery 
experiences, vicarious experiences, verbal persuasion and physiological/emotive states 
(Ponton et al. 2001, 249). Ponton et al. point out that curricular education is typically 
designed to develop self-efficacy through mastery experiences (2001, 249). 

Mastery experiences refer to the performance of past activities and the resulting 
observances regarding whether the performances were successful or not. A noteworthy 
observation from this research is that if the individual perceives that he or she was 
successful in an activity, then the individual will develop a self-perceived sense of 
capability in being able to perform the same, or a similar activity in the future (Ponton et 
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al. 2001, 249). In the process of designing mastery experiences, two important aspects to 
consider are: what exactly do we want students to master; and how are we going to let 
them know? (Ponton et al. 2001, 249). After deciding the important skills a practicing 
engineer should master, we must also tell students why the chosen skills are important 
and develop assessment measures that reinforce the value of the desired skill (Ponton et 
al. 2001, 249). In regards to curriculum development. Ponton et al., stress that the 
important point to consider is that a decision should be made concerning the specific 
skills (behavioral or cognitive) that are important to a practicing engineer, and then create 
mastery experiences that will enhance a student’s sense of competency in performing the 
skills (2001, 249). 

A vicarious experience refers to the development of self-efficacy in an activity 
after observing similar persons engaging successfully in that same activity (Ponton et al. 
2001, 249). Vicarious experiences are used to augment self-efficacy when students see 
similar others performing desired skills. Finally, verbal persuasion refers to developing 
self-efficacy when important referents tell a person he or she has the capability to 
successfully perform an activity (Ponton et al. 2001, 249). 

1. Cognitive Motivation 

Studies have shown that “self-beliefs of efficacy have a large role in the self¬ 
regulation of motivation and that most of human motivation is cognitively generated” 
(Bandura 1993, 128). For example, students who motivate themselves do so by forming 
beliefs about what they can do; they set personal goals and then plan a series of tasks 
aimed to accomplish the goal(s). Three theories have been postulated by Ponton et al. that 
describe the aforementioned mechanisms of cognitive motivation. These are goal theory, 
expectancy value theory (EVT) and attribution theory. 

Ponton et al. note, “The basic premise of goal theory is that when people adopt 
performance goals that they wish to accomplish, motivation is induced by the 
discrepancy between their current and future desired (i.e., goal) states” (2001, 248). 
Recognizing the discrepancy between present and future states is the source of motivation 
towards reaching the goal. As Ponton et al. further explain, “A challenging goal provides 
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a greater discrepancy and as a result, greater motivation because of the anticipated self- 
satisfaction one hopes to attain from achieving the stated goal” (2001, 248). Research 
suggests that goals should be challenging, but not perceived to be beyond one’s capability 
(Ponton et al. 2001, 248). Distal goals that are considered to be beyond one’s capability 
may be achieved if proximal sub-goals are set within one’s scope of self-efficacy (Ponton 
et al. 2001, 248). As such, self-efficacy plays an important role in goal theory (Ponton et 
al. 2001, 248). 

Goal proximity influences motivation in that while distal goals provide a general 
(but weak) motivation for performance, proximal sub-goals that provide immediate 
indicators of success and support the ultimate distal goals, provide a greater source of 
motivation. Accomplishing sub-goals can enhance self-efficacy, thereby increasing an 
individual’s commitment to the overall, distal goal. Thus, the theoretical findings suggest 
that distal goals for students should be apportioned into sub-goals that motivate behaviors 
“leading to the accomplishment of the sub-goals and ultimately the distal goal itself’ 
(Ponton et al. 2001, 248). 

Expectancy value theory (EVT) describes the form of cognitive motivation 
whereby individuals engage in “activities to accomplish goals because of the perceived 
value associated with their likely outcomes” (Ponton et al. 2001, 248). EVT holds that the 
“greater the expectation that a particular behavior will lead to a valued outcome, the 
greater is the motivation to engage in that behavior” (Ponton et al. 2001, 248). EVT also 
holds that the presence of self-efficacy facilitates the process (Ponton et al. 2001, 248). 
Ponton et al. suggest that an important mediating factor self-efficacy includes in relation 
to an individual’s accomplishments is perseverance (2001, 248). Eor instance, their 
research provides an example whereby even if a student perceives that he or she has the 
required capabilities to perform engineering coursework, the student may not fully delve 
into the “engineering curriculum due to a feeling of self-inefficacy with regards to 
sustaining the effort required to complete the program” (Ponton et al. 2001, 249). 

Attribution theory “describes the motivating influences of how an individual 

attributes causes of past accomplishments and failures thereby affecting future choices of 

behaviors” (Ponton et al. 2001, 249). According to attribution theory, if an individual 

17 



attributes the success of a past performance to hard work or failure to a lack of effort, 
then that same individual is more likely to engage in a similar performance in the future 
as the need arises (Ponton et al. 2001, 249). As such, effort attributions are mediated 
through the self-efficacy mechanism (Ponton et al. 2001, 249). 

Effort attribution “refers to how an individual attributes the causes of the effort 
expended in performing a chosen behavior, for example, high effort is indicative of low 
ability” (Ponton et al. 2001, 249). Effort attributions are based on whether ability is 
understood as being of “static or dynamic proportions” (Ponton et al. 2001, 249). If 
ability is understood to be static and temporarily stable, then a high level of effort is 
understood as an indicator of low self-efficacy. Conversely, if ability is understood to be 
dynamic and capable of influence, then ability is enhanced with effort alongside a parallel 
increase in self-efficacy (Ponton et al. 2001, 249). 

Cognitive motivation also works retrospectively, as when an individual reflects on 
the possible causes of successes, failures and effort—forming such evaluations can 
influence future actions (Ponton et al. 2001, 248). Eor instance, an individual may 
interpret through an activity that causes stress (either emotional or physical) that a lack of 
the essential capability required for success is present (Ponton et al. 2001, 249). Ponton et 
al., provide a case study example to describe how physiological/emotive arousals can 
affect self-efficacy. Eor instance, when a student is engaged in a stressful assignment, the 
student may interpret induced stressors as indicators that the capabilities needed to 
successfully complete the assignment are lacking (Ponton et al. 2001, 250). Thus, 
physiological/emotive arousals supply self-efficacy information through the 
interpretations of the reactions (Ponton et al. 2001, 249). 

Ponton et al. conclude their research on the motivational constructs of self- 
efficacy by suggesting that because of the important mediating influence self-efficacy has 
been observed to have on self-motivation, an educator should understand how the trait 
can be increased so that such motivating principles can be used in the design of effective 
instructional strategies (2001, 249). 
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E. SELF-EFFICACY IN COGNITIVE DEVELOPMENT AND 

FUNCTIONING 

According to Bandura, “Conceptions of ability affect thought processes and 
performance attainments through the self-efficacy mechanism” (1993, 121). Human 
functioning is influenced by the beliefs individuals have on how their ability can evolve 
over time (Bandura 1993, 121). For instance, individuals who consider ability as a 
biologically decreasing aptitude with aging are likely to infer poor performance as 
indicative of declining capability. And, they do little to utilize their capabilities. Whereas, 
individuals that perceive ability as a skill that can be developed and practiced, accomplish 
higher achievements (Bandura 1993, 121). 

At this point in the literature review, it is noteworthy to provide an overview on 
theoretical findings of self-efficacy and cognitive development and functioning. One such 
finding is that perceived self-efficacy exerts its influence through four major processes: 
cognitive, motivational, affective and selection (Bandura 1993, 117). 

I. Cognitive Processes 

Research on self-efficacy has shown that it contributes to cognitive development 
and functioning (Bandura 1993, 117). Cognitive processes, similar to cognitive 
motivation discussed earlier, require a strong sense of self-efficacy for one to remain task 
oriented when faced with pressing situations and failures that have social repercussions 
(Bandura 1993, 120). For instance, individuals who perform poorly on tasks may do so 
because they lack the requisite skills, or have the requisite skills, but lack the sense of 
efficacy to use the skills effectively (Bandura 1993, 118). Relative to this theory is the 
conception of ability—a belief system whereby how people regard ability affects their 
cognitive functioning. For instance, some children consider ability as an acquirable skill 
that can be increased by gaining knowledge and competencies. This cognitive process 
leads to a functional-learning goal, in which they seek challenges that provide 
opportunities to expand their knowledge and competencies and consider errors as 
opportunities to learn from their mistakes. Other children see ability as an inherent 
capacity. For them, performance is diagnostic of their inherent intellectual capacities. As 
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Bandura states, “deficient performances carry high evaluative threats that they lack basic 
intelligence.” That is why, they prefer tasks that reduce errors and demonstrate 
proficiency “at the expense of expanding their knowledge and competencies.” 
Furthermore, this acquirable skill view allows for a task-diagnostic focus aimed at honing 
one’s competencies and mastering challenges (Bandura 1993, 120). 

In reference to cognitive processes, self-efficacy seems to have a particular 
mediating effect on writing skills and memory. According to Bandura, “Enhancement of 
perceived writing efficacy by instruction raises, through different paths of influence, 
perceived self-efficacy for academic activities, personal standards for the quality of 
writing considered self-satisfying and academic goals and attainments” (1993, 136). 
Similarly, higher cognitive effort has been shown to produce better memory performance 
(Bandura 1993, 121). 

2. Motivational Processes 

The ability to practice self-influence by personal challenge and evaluate reaction 
to an individual’s achievements provides a major cognitive tool of motivation. Behavior 
is motivated and guided by cognized current goals. Research indicates that precise, 
challenging goals improve and sustain motivation (Bandura 1993, 130). 

3. Affective Processes 

According to Bandura, “People’s beliefs in their capabilities affect how much 
stress and depression they experience in threatening or difficult situations, as well as their 
level of motivation” (1993, 132). This affective processing is the emotional mediator of 
self-efficacy belief. 

4. Selection Processes 

Self-efficacy can influence the decisions individuals make by affecting their 
selection of activities and environments. For instance, “people avoid activities and 
situations they believe exceed their coping capabilities. Yet, they readily opt for 
challenging activities and select situations they judge themselves capable of handling” 
(Bandura 1993, 134). Research has suggested that the higher an individual’s self-efficacy 
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within a domain, the more career options he or she envisions as possible and the greater 
the interest he or she demonstrates in the chosen career (Bandura 1993, 135). Similarly, 
high self-efficacy mediates the process whereby an individual can better prepare 
himself/herself educationally for various occupations along the chosen career path 
(Bandura 1993, 135). One cannot underestimate the lasting impact these findings suggest, 
which is that high self-efficacy within an occupational domain can positively influence 
one’s “staying power and success in difficult occupational pursuits” (Bandura 1993, 135). 

F. SELF-EFFICACY IN THE DEVELOPMENT OF TECHNICAL AND 

PROFESSIONAL SKILLS 

The literature review suggests that there are three categories of skills needed by 
leaders to be successful: conceptual, human and technical (Moore and Rudd 2005, 69). A 
research model, similar to our own, supports this theory by way of results of a survey 
instrument that was designed with 39 competencies. Factor analysis results of the 39 
competency items revealed nine broad competency areas: four represented conceptual 
skills, three represented human skills and two represented technical skills (Moore and 
Rudd 2005, 69). 

Research on leadership competencies has suggested that conceptual competencies 
are the most important, followed by human competencies, and finally with technical 
competencies as the least important asset in leadership KSAs (Moore and Rudd 2005, 
69). Conceptual skills are perhaps the most important at top management levels where 
policy decisions, long term planning, and broad scale actions are required (Moore and 
Rudd 2005, 69). Conceptual skills involve being able to see both what is going on within 
an entire organization and how different components of the organization interact and 
depend on one another (Moore and Rudd 2005, 26). 

Human competencies are defined as the “people skills” or the ability to work 
effectively as a group member and to build cooperative effort within the team being led 
(Moore and Rudd 2005, 69). It is noteworthy to mention here that the NFS’s survey 
instrument and SE competency model encompass such human competencies in the form 
of the “teamwork” domain. Moore and Rudd state that the presence of human skills is 
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important throughout all management levels (2005, 69). Ideally, human skills are shown 
in how a leader perceives and behaves towards those around him or her, including 
superiors, peers, and followers. More importantly, it must be demonstrated in every 
action of the leader (Moore and Rudd 2005, 69). This is why the NFS’s competency 
model maps leadership KSAs throughout a system engineer’s career. 

Goleman extends the theoretical framework on leadership competencies from 
purely technical skills and cognitive abilities to include that of emotional intelligence (El) 
(Moore and Rudd 2005, 69). He suggests that there are five components to El: self- 
awareness, self-regulation, motivation, empathy and social skill (Moore and Rudd 2005, 
69). Goleman includes El as a set of leadership skills because he sees it as a 
distinguishing competence of senior leaders (Moore and Rudd 2005, 69). A noteworthy 
research finding is that Goleman found El to be twice as important as the other leadership 
skills when applied to all levels of jobs within the organizational hierarchy. He also found 
that El (rather than conceptual skills) explained 90 percent of the difference between the 
effectiveness of lead performers and average senior level leaders (Moore and Rudd 2005, 
69). 

An agricultural study conducted by Moore and Rudd used data on land-owning 
individuals to assess the importance and level of six leadership skill areas (2005, 68). 
Their research cites the notion that while “leadership skills can be taught and learned, 
skilled leaders continue to be in short supply” (Moore and Rudd 2005, 68). To be 
successful as a leader, an individual must understand how to motivate a group of 
individuals (i.e., students, or entry level employees) towards a common goal (i.e., the 
desired future state) (Ponton et al. 2001, 247). 

Robbins, Bradley and Spicer (2011) designed a leadership competency 

assessment instrument for healthcare administration, with 52 items classified as technical 

skills, industry knowledge, analytic and conceptual reasoning or interpersonal and El 

(Moore and Rudd 2005, 70). Their findings conclude that industry knowledge should be 

included as a domain of skills due to the complex nature of the health care industry 

(Moore and Rudd 2005, 70). Industry knowledge is defined as having expansive 

knowledge of the industry and the organization. It is “essential to creating the broad 
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outlook needed by leaders to produce an organizational vision and strategies to 
accomplish that vision” (Moore and Rudd 2005, 70). As such, one could argue that this is 
another domain of competencies within engineering, and one that should be further 
studied to see if industry knowledge affects self-efficacy. 

Moore and Rudd suggest that technical skills are associated with understanding 
and are the skills, methods, processes, procedures and techniques required to complete 
specific activities (2005, 69). Their findings suggest that technical skills are more 
important at the lower levels of administration, and that specifically, as a leader moves up 
in the organizational hierarchy, he or she relies on the technical skills of followers more 
than on his or her own technical skills (Moore and Rudd 2005, 69). Survey research led 
Moore and Rudd to find that survey participants perceived technical skills to be the least 
important of the skills set expected of leaders (2005, 75). Their finding that technical 
skills are perceived as less important than other leadership competencies is consistent 
with literature that reports the amount of technical skills required by leaders decreases the 
higher in the organizational hierarchy the leaders go, because of the tendency to delegate 
technical tasks to lower level staff (Moore and Rudd 2005, 69). Finally, Moore and Rudd 
recommend that competencies within the communication skill area be included in 
leadership and professional development programs (2005, 70). 

G. CHAPTER SUMMARY 

The literature, thus, suggests that domain self-efficacy aids in the development of 
an individual’s initial interest in a corresponding career area, influences his/her 
motivation to pursue a career within the field, as well as the behavior towards the 
selection of relevant career paths, and then supports the higher levels of both academic 
performance and persistence in the field. Pertinent to NFS’s research on self-efficacy 
within engineering is foundational research performed by Lucas et al. via a survey model, 
which found that the role of self-efficacy continues to be important past education and 
training, when individuals have entered their careers, as the sociopsychological construct 
can be used to predict better work performance. Similar research by Joshi et al. provides 
another example of a successful attempt to measure self-efficacy as a latent psychological 


23 



trait using a survey instrument. Elaborating on these and other research findings that 
suggest that high levels of self-efficacy are indicative of the development of mature 
cognitive and affective processes (which can be used to predict proficient work 
performance), NFS has undergone research to identify, via a competency model and the 
SE Competency Survey, relevant engineering KSAs and the developmental stages of such 
cognitive and affective processes. In closing, NFS stipulates that self-efficacy is a useful 
measure of academic and work performance, and provides research to support the 
assertion on the usefulness of the construct as a reliable indicator of career performance. 
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III. MODEL BASED RESEARCH ON TECHNICAL SKILLS, 
NON-TECHNICAL SKILLS AND SELE-EEEICACY 
AS REGULATORS OE COGNITIVE AND 
AEEECTIVE PROCESSES 

A. INTRODUCTION: COMPETENCY MODEL DEVELOPMENT 

Relative to NFS’s research was the role of competency modeling, defined as the 
“activity of determining the specific competencies that are characteristic of high 
performance and success in a given job” (LaRocca 2013). To understand the relationship 
competency modeling has with the identification of the KSAs required of engineers, NFS 
developed a systems engineering career development competency model. Referred to in 
research and literature as the competency of the profession of systems engineering 
(COMFOSE), the model highlights an aggregate of the core technical, technical 
management and professional (i.e., non-technical) KSAs researched from the current 
hiring manuals of various naval and engineering enterprises (White 2014, 4). The NFS 
team feels that systems engineers need the professional competencies at higher 
proficiency levels in the earlier stages of a career than disciplinary engineers (Whitcomb, 
Khan and White 2013). 

NFS believes the competency model can be used to identify and evaluate KSAs 
required to develop self-efficacy across various engineering domains. As such, it can be 
used by universities to ensure training and education meet the needs of industry. Two key 
objectives of the COMFOSE model were to: develop an approach and methodology to 
obtain baseline information needed for SE competency model development (for example 
the related KSAs, relevant education, training, and experience required) and to define 
career paths for systems engineers (jobs, assignments, and timing). 

Research on self-efficacy has suggested that an individual’s level of self-efficacy 
is a motivational predictor of how well a person will perform at almost any endeavor, and 
can be utilized and developed to leverage performance enhancing advantages (National 
Security Eeadership Eoundation n.d.). Thus, it can be understood that self-efficacy serves 
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as a regulator across a broad range of functioning, as the key determinant for human 
cognitive/affective, motivational and decision-making processes (National Security 
Leadership Foundation n.d.). 

B. DEFINING SE COMPETENCIES: CLASSIFICATION BASED ON 

AFFINITY 

The NPS research team developed a holistic approach to the design and structure 
of COMPOSE as a systems engineering competency model. Several competency models 
were used to verify the COMPOSE model in an effort to combine skills from different 
sources to generate a complete scope of SE KSA. Collectively, the competency models 
were mapped to the 41 competencies of the DAU SE Competency Model to form the 
basis of the competency list for the COMPOSE model. The competency classification 
verification process, based on affinity, was achieved by mapping KSA to proficiencies 
within the competencies using key words from Bloom’s taxonomy, and then correlating 
the KSA to a specific cognitive or affective category. In the current version of the 
COMPOSE model there are 27 technical and technical management competencies, 14 
professional competencies, and 2,914 KSAs along three notional skill levels, discussed in 
the next section (Whitcomb, Khan and White 2013). 

C. COGNITIVE AND AFFECTIVE LEARNING DOMAINS: BLOOM’S 

TAXONOMY 

Of noteworthy significance to the purpose of this thesis is the research conducted 
by White on the development of the COMPOSE model, and its identification and 
evaluation of the cognitive and affective domains associated with SE KSA (2014). The 
COMPOSE model categorizes KSAs in the form of Bloom’s taxonomy into either the 
cognitive or affective domain, for each KSA in both technical and non-technical SE 
competencies. As such, the COMPOSE model allows for investigation into the break out 
into cognitive and affective processes (reference Eigures 1-3). As White’s research 
(reference Eigures 2 and 3) reflects, 67 percent of the COMPOSE model’s KSAs are 
mapped into the cognitive domain, with the remaining 33 percent being in the affective 
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domain (2014, 18). As reflected in Figures 1-3, the COMPOSE model illustrates how the 
specific SE related competencies that are characteristic of high performance and success 
in SE related jobs can be apportioned. With 33 percent (Eigure 1) of the KSAs being 
grouped in the affective domain, this illustrates that this is important to the development 
of a SE. In addition, the NPS research provides weight to current research findings that 
suggest that affective learning domains, such as to value and respond (Eigure 3), are 
“important constructs today’s students must possess” and are essentially a deciding factor 
to the successful career achievement of lead performers (Joshi et ah, 2012; Moore and 
Rudd 2005). 

Eurthermore, the analysis illustrates how cognitive and affective levels span the 
entire domain of SE (Eigures 2 and 3). In COMPOSE, every level within each learning 
domain is represented, so as such, it is important to include all of the levels in aspects of 
assessing self-efficacy, as NPS does so via the SE Competency Survey. If one can identify 
his/her level of self-efficacy in each of the SE domains, and by doing so, also quantify 
KSA associated with the cognitive/affective learning domains, it is reasonable to assume 
one can then also gain an understanding of where he/she is at in being able to perform 
domain specific tasks. This is significant as research indicates an individual’s perceived 
level of self-efficacy greatly influences his/her ability to achieve domain specific tasks 
(Eucas and Cooper 2004, 12). 
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COMPOSE Cognitive and Affective 
Breakout 



■ Cognitive 

■ Affective 


Figure 1. COMPOSE Model’s Cognitive and Affective Category Breakout 

(from White 2014, 54). 



■ Remember (R) 

■ Understand (U) 

■ Apply (AP) 

■ Analyze (AN) 

■ Evaluate (EV) 

■ Create (C) 


Figure 2. Bloom’s Cognitive Levels within the COMPOSE Overall Model 

(from White 2014, 55). 
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Figure 3. Bloom’s Affective Levels within the COMPOSE Overall Model 

(from White 2014, 56). 


In reference to the COMPOSE model, the cognitive domain includes knowledge, 
critical thinking and the development of intellectual skills (White 2014, 15), whereas, the 
affective domain deals with the emotions, feelings and attitudes of systems engineers. 
Affective KSAs describe growth in awareness, attitude, emotion, changes in interest, 
judgment and the development of appreciation (White 2014, 16). While most educators 
do not include the affective learning domain in the development of their curriculum. 
White states, “these outcomes are especially critical to the success of systems engineers 
as [they] lead systems projects, negotiate outcomes with a diverse group of stakeholders, 
make value judgments and must have the ability to deliberately take the systems 
perspective” (2014, 16). 

The development of cognitive and affective domains is categorical and 
hierarchical, with increasing degrees of complexity (Clark 1999). Bloom’s theory of 
categorical and hierarchical domains of learning is mirrored in the COMPOSE model, as 
it utilizes Bloom’s taxonomic approach to aid in the description and classification of 
KSAs required of engineers. 
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To illustrate, as an engineer progresses from various levels of experienee, it is 
assumed he/she improves in knowledge, skills and abilities related to the field. This is 
signifieant to mention as researeh suggests that skill development is a continuous process, 
and as knowledge and procedures continue to be compiled, more “elegant task strategies 
emerge” (Kraiger, Ford and Salas 1993, 315). 

The NFS competency model takes the taxonomic approach one step further by 
providing an applicable method to document and assess at which points, and at which 
depth, should an engineer demonstrate particular KSAs within the 27 core technical and 
14 non-technical competencies identified. 

D. SELF-EFFICACY AND KSAS: PREDICTORS OF WORK 

PERFORMANCE 

Research on self-efficacy within engineering suggests that while work experience 
will facilitate the acquisition of considerable knowledge and the learning of new skills, it 
also has the advantage of enabling engineers to carry out work tasks where their 
performance will be assessed both by workplace professionals and the engineers 
themselves (Lucas et al. 2009, 741). 

As previously described, the 41 competencies included in the COMPOSE model 
can be viewed as the core technical and non-technical competencies vital to the 
performance of an engineer. Select KSAs represented in the Systems Engineering 
Competency Survey allow a means whereby competencies defined by the COMPOSE 
model can be applied in a practical means by workplace professionals and engineers to 
assess performance. As such, the survey instrument aids in the assessment a person 
within a supervisory role can make of where an engineer is at in his/her respective field in 
terms of performance, and it also offers a means by which an engineer can measure 
his/her own level of self-efficacy in performing engineering functions. This is significant 
as research linking self-efficacy to performance indicates it “continues to be important 
once individuals have entered their careers, predicting better work performance” (Eucas 
et al. 2009, 740). 
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A survey of 114 research reports on self-efficacy and work-related performance 
show self-efficacy reliably “predicts higher levels of work performance across a wide 
variety of types of organizations and activities" (Stajkovic and Luthans 1998). For 
instance, in business it has been observed that real-estate agents with higher self-efficacy 
in their competencies have higher sales, and otherwise outperform peers with lower self- 
efficacy. Similarly, “causal inferences about the effect of self-efficacy on work 
performance is supported by experimental research that uses random group assignment 
and the manipulation of self-efficacy” (Lucas et al. 2009, 740). Research also suggests 
that self-efficacy links with cognitive/affective behaviors such as innovation and 
opportunity recognition (Ardichvili, Cardozo and Sourav 2003). 

1. Role of Cognitive and Affective Competencies in Job Performance 

Research suggests that an individual’s level of self-efficacy is a motivational 
predictor of how well a person will perform at almost any endeavor, and can be utilized 
and developed to leverage its performance enhancing advantages (National Security 
Leadership Foundation n.d.). An individual’s belief in their level of self-efficacy (or 
perceived self-efficacy) greatly influences their ability to achieve domain specific tasks. 
In turn, self-efficacy serves as a regulator across a broad range of functioning, as the key 
determinant for human cognitive, affective, motivational and decision-making processes 
(National Security Leadership Foundation n.d.). 

Research on the role of cognitive learning strategies suggests that individual 
differences exist in the extent to which knowledge can be accessed or applied more 
rapidly or fluidly (Kraiger, Ford and Salas 1993, 311). One theory relating the role of 
cognitive learning strategies suggests that through continued practice, complex behaviors 
are internalized. The greater the internalization, the more cognitive resources are 
available for higher level functions or strategy development (Kanfer and Ackerman 
1989). In context to how strategy development relates to the NFS SE competency model 
and survey instrument, it is the “broad range of mental activities [i.e., KSAs] that 
facilitate knowledge acquisition and application” (Prawat 1989). Thus, as knowledge is 
acquired, it facilitates the application, or performance of work. 
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E. CONCLUSION 


The analysis aimed to look at the cognitive and affective learning processes of 
each KSA for the professional development SE competencies identified through the 
analysis of various engineering development models. Skills are a composite of abilities, 
techniques, and knowledge. They are developmental and incremental, and reference- 
based for a desired application, in our case, systems engineering (Clark 1999). The 
COMPOSE model, which serves as a guide to identify and evaluate the KSAs required to 
be a proficient engineer, identifies the technical and non-technical competencies across 
three notional proficiency levels within each competency. Eurthermore, the COMPOSE 
model includes 14 non-technical competencies in the framework of professional skills a 
leader (especially one within an engineering discipline) should develop, and offers an 
analysis of how the competencies can be decomposed into individual KSAs. The thesis 
argues that self-efficacy, through competency modeling and surveying, can be enhanced 
within an individual, and, as a predictor of work performance, can aid in the development 
of proficient, highly skilled engineers. 

F. CHAPTER SUMMARY 

The chapter focused on describing NPS’s efforts on applying model based 
research of technical and non-technical skills and self-efficacy to develop a competency 
model. The chapter began with a description of the process whereby NPS developed a 
systems engineering career development competency mode—COMPOSE. The chapter 
also detailed how the COMPOSE model maps relevant cognitive and affective processes 
to SE specific KSAs. The NPS research team contends that the competency model can be 
used to identify and evaluate KSAs required to develop self-efficacy across various 
engineering domains. And, as such, the model can be used by universities to ensure 
education and training meet the needs of industry. The chapter concluded with the 
argument that self-efficacy, through competency modeling as the basis for surveying, can 
aid in the development of proficient, highly skilled engineers. 


32 



IV. MEASURING SELF-EFFICACY: DEVELOPMENT OF A 
SELF-EFFICACY MEASUREMENT INSTRUMENT AND 
ANALYSIS OF NPS STUDY 

A. HOW DO SELF-EFFICACY ITEMS IN THE SURVEY INSTRUMENT 

RELATE TO THE COMPOSE MODEL? 

In the previous chapter, the implications for possible applications of the 
COMPOSE model as a source of self-efficacy related KSAs was discussed. This chapter 
is an extension of the research previously discussed, and aims to demonstrate how the 
survey instrument developed by the SE department at NPS can be used to 1) aid in the 
development of competent systems engineers and 2) be used as a tool to predict career 
performance. The Systems Engineering Competency Survey is a survey instrument 
initially designed with a composite of questions aimed to measure the level of self- 
efficacy perceived by entry level systems engineers. The self-efficacy items in the The 
Systems Engineering Competency Survey relate to the COMPOSE model in that they 
pose the cognitive and affective KSAs measures as questions. In other words, the survey 
items relay the KSAs exhibited in the model as measures of performance. 

B. NPS SURVEY INSTRUMENT: BACKGROUND AND METHODOLOGY 

The primary method utilized to measure self-efficacy in systems engineering 
students was via the Systems Engineering Competency Survey. The survey instrument had 
65 items meant to measure the knowledge, skills and abilities that are directly related to 
meeting educational objectives one would expect of a master’s systems engineering 
program. The survey items were developed by subject matter experts, using information 
collected during the development of a DOD competency model for the domain of 
systems engineering, and as such are believed to have face validity. 

The survey items were written to measure self-confidence in performing systems 
engineering related tasks that included understanding the limits of systems engineering 
technology and techniques (critical thinking), translating user requirements into technical 
requirements (systems engineering), leading a team of technical professionals 

(teamwork), and being able to track and monitor the progress of a project (project 
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management). The Systems Engineering Competency Survey asked students to rate their 
level of self-efficacy on a linear scale of 0 to 100, with 0 representing “no confidence” 
and 100 being ‘complete confidence’ in being able to perform the skills. The expectation 
was that the survey items constituted tasks that graduates would be able to perform if 
they were to be successful in systems engineering work. 

The survey was delivered in a pre- and post-mode to two consecutive SE cohorts 
(categorized as RT19 and RT19A, respectively). The RT19 pre-rendition of the SE 
Competency Survey was deployed to an SE cohort in 2011 during the middle of the 
students’ first quarter. The RT19A cohort was given the pre-survey at the beginning of 
their program, in 2012. Then, each of the RT19 and RT19A post-surveys were 
administered towards the end of the each of the cohorts’ program. 

1. SE Competency Survey: Questionnaire Analysis Using Factor 
Analysis and Cronbach’s Alpha 

Eollowing an example set by Eucas et ah, NFS used the statistical technique of 
factor analysis (EA) to test for underlying dimensions of the self-efficacy items 
embedded in the survey (2009, 744). As defined by Ericker, Kulzy and Appleget: 

factor analysis is a method for identifying latent traits from question-level 
survey data. It is useful in survey analysis whenever the phenomenon of 
interest is complex and not directly measurable from a single question. In 
such situations, it is necessary to ask a series of questions about the 
phenomenon and then appropriately combine the resulting responses into a 
single measure or factor. Such factors, then, become the observed 
measures of the unobservable or latent phenomenon. (2012, 1) 

If the EA results indicate that an established scientific reliability goal is achieved for each 
survey item, “the latent variables will have useful and interpretable meanings that provide 
additional insight into the characteristics of the populations being studied” (Ericker, 
Kulzy and Appleget 2012, 2). 

Eor the purposes of our study, the latent phenomenon was defined to be the 
cognitive trait(s) of self-efficacy to perform each competency. We used the technique of 
EA as a method to extract and label each latent competency, which we then grouped into 
one of the four systems engineering domains. And, again following an example set forth 
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by Lucas et al., the criteria for establishing the scientific reliability goal was that the FA 
result for each survey item must be .6 or higher on its “primary component,” and less 
than .4 on any other component. We define primary component as the domain we 
intended the competency to fall under. So, for instance we intended for each competency 
to fall under project management, teamwork, systems engineering or critical thinking. 
Factors were grouped together based on intended domain and the FA value obtained. 

The purpose of the factor analysis was to determine which items of the Systems 
Engineering Competency Survey, if any, could be considered to be reliable indicators of 
self-efficacy, within one of the four specific domains of systems engineering. The factor 
analyses were run twice, once for each pre-survey rendition of the SE Competency 
Survey. The method of FA allowed the NFS research team to delineate and group the 
competencies, based on the value assigned to each competency from the survey results. 
As stated above, competencies that loaded with a value of .6 or higher in the primary 
component (domain) were retained for further analysis. Competencies less than .4 were 
excluded from the analysis to strengthen the identification of the four specific self- 
efficacy components. 

Cronbach’s alpha, a statistical measure used to test the validity and reliability of 
survey data, was then run for each of the 65 RT19 and RT19A survey items, and again for 
each survey set as a whole. Cronbach’s alpha was observed to be .989 and .985 for the 
RT19 and RT19A surveys, respectively—suggesting that overall, the survey items have a 
relatively high internal consistency. 

2. Rotated Factor Analysis 

The responses of each pre-rendition of the RT19 and RT19A survey were used in 
the factor analysis to test for underlying dimensions within the self-efficacy survey items. 
In FA, the relationship of the factors to each other is determined by a rotation technique 
selected during the analysis. Following an example set forth by Lucas et al., we chose to 
rotate our FA results using the statistical rotation technique referred to as Varimax. In 
FA, the rotated component matrix displays the loadings, or value groupings, of variables 
on factors. Variables are grouped based on “high loadings” (.6 or higher) and similar 
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values across components. If the factor axes are rotated, the loading of a variable on one 
factor is maximized while its loading on the other factors is minimized, thereby making 
the factor structure easier to interpret (Daniel 2013). For the context of this paper, we 
define the phrase “loaded on” or “high loadings” to refer to the way in which FA results 
are displayed. Survey items with a value of .6 or higher are said to “load” strongly - 
meaning they can be assumed to reliably measure the latent construct one is examining. 
FA results display survey items within matrix columns. These columns are suggested as 
being domains. 

The significance of using the Varimax rotation method after performing FA is that 
the Varimax rotation attempts to simplify the columns of the factor matrix, achieving the 
maximum simplification of component identification. This also results in factors that are 
independent of each other (Daniel 2013). For the purposes of our study, this was 
important to do as we aimed to 1) test the hypothesis that self-efficacy can be measured 
and improved upon by a rigorous education and training and 2) identify reliable measures 
of self-efficacy in each of the four systems engineering domains. 

When the responses were subjected to rotated factor analysis, 11 components 
emerged both times, with the four expected domains of systems engineering (i.e., critical 
thinking, systems engineering, teamwork or project management) emerging as major 
components. Again, the component selection was determined using competencies that 
loaded on a .6 or higher within the intended systems engineering domain and excluding 
the competencies that loaded strongly (.6 or higher) on more than one domain, or scored 
a value lower than .4. Student data for each cohort set was then subjected to two sample 
t-tests, to examine if any statistically significant difference was observed between the 
mean pre- and post-self-efficacy scores. 

C. GENERAL RESULTS 

Using the factor analysis results of the survey items tested against each other, it 
was observed that some survey items loaded on more than one component—thereby 
indicating that the items may need to either be re-evaluated (so as to measure one specific 
domain) or be eliminated from future renditions of the survey. A possible follow on 
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procedure, now that factor analysis has been performed twice, could be a cross analysis 
of each of the SE Competency Survey factor analysis results to determine which, if any, 
of the survey items can be used for further survey analysis. Taking into consideration 
each of the factor analysis results, we may re-combine certain survey items or re-write a 
survey to measure self-efficacy in the SE domains of teamwork, project management, 
systems engineering and critical thinking. 

It is noteworthy to mention that the data reflected in Tables 1 and 2 are a subset of 
the 65 survey items that were originally presented to the students in the pre- and post¬ 
surveys. The survey items represented are those that, through factor analysis, were 
observed to load on a single, distinct SE domain. 


Table 1. Pre to Post Education Change in Eevels of Self-Efficacy, AY 2011 

Cohort 


A. Self-efficacv: Current skill levels in 

Teamwork 

Percent a’ 
all res| 

vg. across 

3onses 

Response changes from 
pre- to post-event survey 
and sign test 

Response rate at post-test, 17/17=100% 

Pre¬ 

education 

Post¬ 

education 

N 

Percent 

increase 

T value 

Help two team members resolve serious 
personal differences so they can work 
effectively with the team. 

74 

79 

17 

5 

-0.955 

Help make sure your teammates deliver 
on the schedule that they agreed to 
support. 

76 

78 

17 

2 

-0.457 

Talk with your team members for the 
purpose of assessing their interests and 
skills in order to ensure needed team 
capabilities are included. 

75 

79 

16 

4 

-0.716 

Identify important personal relationships 
that must be considered to complete a 
project. 

69 

74 

14 

5 

-0.920 

Determine an operating plan once all team 
members understand their interests and 
skills. 

66 

75 

14 

9 

-1.9316* 

Get your team members at project start to 
make personal commitments to their 
responsibilities. 

69 

70 

14 

1 

-0.143 
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Give constructive criticism to a poorly 
performing team member that helps them 
improve their performance. 

Facilitate a team to reach agreement when 
there had been conflicting views about 
what steps should be taken next. 


B. Self-efficacy : Current skill levels in 

Critical Thinking 


Examine the statement of a problem to 
identify its key elements. 

Identify and, if necessary, challenge the 
assumptions in a problem statement. 

Examine what is known and unknown 
about a problem to create a logical 
argument for the best solution. 

Read a policy paper and identify any 


14 2 


-0.434 


- 0.212 


Percent avg. across 
all responses 


Response changes from 
pre- to post-event survey 
_and sign test_ 


Pre¬ 

education 

Post¬ 

education 

N 

Percent 

increase 

T value 

72 

76 

17 

4 

-1.005 

73 

71 

17 

-2 

0.527 

68 

74 

17 

6 

-1.102 


assumption that is unsupported by the 
evidence. 

66 

71 

17 

5 

-0.838 

Identify the critical questions of fact that 
are missing from a line of reasoning. 

64 

70 

14 

6 

-1.020 

Eind an inconsistency in an argument the 
first time you read a document. 

64 

68 

14 

4 

-0.714 

Recognize any logical inconsistencies in a 
presentation by a speaker. 

70 

71 

14 

1 

-0.078 

Judge the strengths and weaknesses of the 

70 

74 

14 

4 

-0.824 


evidence supporting your logic. 


C. Self-efficacy: Current skill levels in 


Percent avg. across 
all responses 


Response changes from 
pre- to post-event survey 
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Use projected warfighting capability gaps 
to estimate the future need for a new 
system. 

60 

74 

17 

14 

-2.9528** 

Translate the system capabilities that you 
have co-developed with stakeholders into 
technical requirements. 

64 

71 

17 

7 

-1.133 

Determine whether a set of system 
requirements are feasible. 

64 

70 

17 

6 

-1.0415 

Create a feasible design that meets the 
system requirements. 

65 

67 

17 

2 

-0.4564 

Pick a technology for a system 
considering that the technology readiness 
level is estimated, and uncertain. 

47 

68 

17 

21 

-3.8973*** 

Recognize when you can reduce the 
complexity of your completed system 
design. 

59 

68 

17 

9 

-2.0062* 

Prioritize the requirements that a future 
customer has for a system, service, or 
construct. 

68 

72 

17 

4 

-0.7142 

Design a system that fits within its desired 
total ownership cost. 

55 

66 

17 

11 

-2.0505* 

Determine what evidence is required as 
data to demonstrate that a system meets its 
requirements. 

61 

71 

17 

10 

-1.9378 

Meet with 20 stakeholders with very 
different needs and determine the design 
that serves them best. 

55 

70 

16 

15 

-2.5056 

Create a system architecture that has 
appropriate elements, attributes, and 
relationships to be able to be used for 
system development and decision making. 

61 

69 

17 

8 

-1.3956 

Formulate a methodology and approach 
for solving a systems engineering problem 
in an unfamiliar subject area. 

62 

70 

17 

8 

-1.6069 

Decompose a problem into its solvable 
parts. 

76 

74 

17 

-2 

0.6378 

Compare a number of proposals and 
identify the one with the strongest line of 
reasoning for how to solve the problem. 

72 

72 

17 

0 

-0.1535 

Listen to 20 stakeholders with very 
different needs and define a set of 
requirements that meet their needs. 

61 

69 

14 

8 

-1.6897 
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Create a rough vision for a design solution 
in an hour for requirements that you have 
just received. 

Know how and when to challenge the 
assumptions for a design. 

Collect and interpret stakeholder needs 
related to a given project. 

Use your technical knowledge to make 
contrihutions to a new design proposal. 

Select the most appropriate initial design 
iteration to meet a given set of 
requirements. _ 

Describe what resources will he necessary 
to conduct tests to demonstrate that a 
system meets its requirements. 


D. Self-efficacy : Current skill levels in 

Project Management 


Develop a time-phased budget for a multi¬ 
year project. 

Estimate quantitatively the likelihood and 
relative impact of events on a 
development project that represent the 
most pressing risks. 

Create a work breakdown structure for 
your project. 

Identify risk events that could have an 
impact on timely development of your 
system. _ 

Recognize the important assumptions that 
need to be tested at the start of a new 
system development project. 

Identify and define the roles and 
responsibilities that will be needed to 
complete a project. 

Determine what measures must be taken 
in a risk mitigation strategy to assess if the 
likelihood or impact have been assuaged 
as development progresses. 


63 

71 

14 

8 

-1.3911 

63 

67 

14 

4 

-0.8112 

69 

72 

14 

3 

-0.6071 

69 

69 

14 

0 

-0.0454 

64 

71 

14 

7 

-1.2259 

58 

71 

14 

13 

-2.968** 


Percent avg. across 
all responses 


Response changes from 
pre- to post-event survey 


Pre¬ 

education 

Post¬ 

education 

N 

Percent 

increase 

T value 

52 

62 

16 

10 

-1.485 

59 

70 

17 

11 

-2.022* 

62 

71 

17 

9 

-1.455 

69 

71 

14 

2 

-0.449 

65 

69 

13 

4 

-1.032 

71 

75 

14 

4 

-1.032 

68 

71 

14 

3 

-0.671 
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Identify in the development schedule 
when tests must be completed to 
demonstrate that a system meets its 
requirements. 

63 

70 

14 

7 

-1.505 

Prioritize events in a risk mitigation 
strategy and ideate corrective action. 

68 

72 

13 

4 

-0.684 


* p<.05; ** p<.01; *** p<.001. The t-test was used to estimate statistical strength of significance using the 
difference of means between the pre- and post-survey results. 


Given a null hypothesis that any change found will be random and equally probable in either direction, the 
t-test calculates the probability that the number of changes is disproportionately in the hypothesized 
direction (one-tail test). 


Table 2. Pre to Post Education Change in Level of Self-Efficacy, AY 2012 

Cohort 


A. Self-efficacy: Current skill levels 

in Teamwork 

Percent avg. across all 
responses 

Response changes from pre- to 
post-event survey and sign test 

Response rate at post-test, 
11/11=100% 

Pre¬ 

education 

Post¬ 

education 

N 

Percent 

increase 

T value 

Ensure team members follow 
agreed upon rules for meetings. 

67 

89 

11 

22 

-5.6903*** 

Help two team members resolve 
serious personal differences so they 
can work effectively with the team. 

59 

81 

11 

22 

-4.2807*** 

Help make sure your teammates 
deliver on the schedule that they 
agreed to support. 

57 

79 

11 

22 

-5.3094*** 

Talk with your team members for 
tbe purpose of assessing tbeir 
interests and skills in order to 
ensure needed team capabilities are 
included. 

55 

76 

11 

21 

-5.0963*** 

Examine two design approaches 
and select the better one for your 
project to present as your 
alternatives to your team. 

51 

78 

11 

27 

-6.3165*** 

Identify important personal 

relationships that must be 

considered to complete a project. 

53 

83 

11 

30 

-6.3555*** 

Help ensure your team members 
understand the goals, scope, and 
deadlines at the project start. 

55 

84 

11 

29 

-6.4187*** 

Get your team members at project 
start to make personal commitments 

50 

76 

11 

26 

-5.4103*** 
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to their responsibilities. 






Give constructive criticism to a 
poorly performing team member 
that helps them improve their 
performance. 

54 

71 

11 

17 

-2.2975* 

Facilitate a team to reach agreement 
when there had been conflicting 
views about what steps should be 
taken next. 

50 

78 

11 

28 

-6.1914*** 

B. Self-efficacy: Current skill levels 

in Critical Thinking 

Percent avg. across all 
responses 

Response changes from pre- to 
post event survey and sign test 


Pre¬ 

education 

Post¬ 

education 

N 

Percent 

increase 

T value 

Analyze opposing positions on an 
issue and compare the quality of 
their evidence. 

59 

77 

11 

18 

_4 199 ^!:^!^^!: 

Examine an engineering problem 
and prepare a written and oral 
statement describing all essential 
elements included in the solution. 

36 

72 

11 

36 

_ 8 , 477 ^^^ 

Evaluate data about an existing 
design problem to define its nature, 
boundaries, and scope. 

40 

73 

11 

33 

-8.435*** 

Eind an inconsistency in an 
argument the first time you read a 
document. 

47 

67 

11 

20 

-4.307*** 

Recognize any logical 

inconsistencies in a presentation by 
a speaker. 

48 

68 

11 

20 

-3.147** 

Identify the critical questions that 
need to be answered before a 
feasible design can be selected. 

39 

73 

11 

34 

-6.148*** 

Judge the strengths and weaknesses 
of the evidence supporting your 
logic. 

45 

75 

11 

30 

-7 

C. Self-efficacy: Current skill levels 

in Systems Engineering 

Percent avg. across all 
responses 

Response changes from pre- to 
post-event survey and sign test 


Pre¬ 

education 

Post¬ 

education 

N 

Percent 

increase 

T value 

Generate alternative system ideas to 
meet a stakeholder need. 

45 

76 

11 

31 

-6.955*** 
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Elicit requirements that a future 
stakeholder has for a system, 
service, or construct using direct 
interaction with them. 

39 

73 

11 

34 

-9.6184*** 

Leverage features or elements of 
existing systems that can inform a 
new design. 

42 

75 

11 

33 

-8.6191*** 

Translate the system capahilities 
that you have co-developed with 
stakeholders into technical 

requirements. 

35 

70 

11 

35 

-7.7072*** 

Create a feasible design that meets 
the system requirements. 

40 

72 

11 

32 

-7.6047*** 

Pick a technology for a system 
considering that the technology 
readiness level is estimated, and 
uncertain. 

33 

68 

11 

35 

-6.1698*** 

Design a system that fits within its 
allocated budget, trading off 
schedule, performance, and 

lifecycle issues. 

37 

68 

11 

31 

-4.5292*** 

Recognize when you can reduce the 
complexity of your completed 
system design. 

39 

68 

11 

29 

-4.6898*** 

Prioritize the requirements that a 
future customer has for a system, 
service, or construct. 

40 

74 

11 

34 

-6.3318*** 

Design a system that fits within its 
desired total ownership cost. 

30 

61 

11 

31 

-4.8887*** 

Determine what evidence is 
required as data to demonstrate that 
a system meets its requirements. 

37 

66 

11 

29 

-6.8608*** 

Tailor a systems engineering 
process for your project. 

31 

76 

11 

45 

-9.3137*** 

Apply functional analysis during 
system design. 

36 

75 

11 

39 

-6.8608*** 

Identify what tests should be 
conducted to create data to 
demonstrate that a system meets its 
requirements. 

36 

68 

11 

32 

-7.4133*** 

Create a system architecture that 
has appropriate elements, attributes, 
and relationships to be able to be 
used for system development and 
decision making. 

29 

70 

11 

41 

-7.4799*'“'* 
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Formulate a methodology and 
approach for solving a systems 
engineering problem in an 
unfamiliar subject area. 

32 

73 

11 

40 

-6.5519*** 

Respond to a change in the 
environment or stakeholder needs 
via modification of the 

requirements baseline. 

32 

61 

11 

29 

-5.2495*** 

Explain tbe ramifications to a 
change in the environment or 
stakeholder needs based on 
modification of requirements. 

33 

65 

11 

32 

-5.6638*** 

Select the most appropriate initial 
design iteration to meet a given set 
of requirements. 

36 

67 

11 

31 

-6.8845*** 

Describe what resources will be 
necessary to conduct tests to 
demonstrate that a system meets its 
requirements. 

38 

68 

11 

30 

6.1492*** 

D. Self-efficacy: Current skill levels 

in Project Management 

Percent avg. across all 
responses 

Response changes from pre- to 
post-event survey and sign test 

Recognize the important 

assumptions that need to be tested 
at the start of a new system 
development project. 

37 

67 

11 

31 

-4.4662*** 

Determine what measures must be 
taken in a risk mitigation strategy to 
assess if tbe likelihood or impact 
have been assuaged as development 
progresses. 

36 

73 

11 

37 

-8.2558*** 

Prioritize events in a risk mitigation 
strategy and ideate corrective 
action. 

41 

78 

11 

37 

-8.0621*** 


* p<.05; ** p<.01; *** p<.001. The t-test was used to estimate statistical strength of significance using the 
difference of means between the pre- and post-survey results. 


Given a null hypothesis that any change found will be random and equally probable in either 
direction, the t-test calculates the probability that the number of changes is disproportionately in the 
hypothesized direction (one-tail test). 

NPS included only those items that were observed to load with a .6 or higher on a 
component, and loaded with items that were similarly intended to measure one of the 
abovementioned domains. And, again, following the model set forth by Lucas et ah, 
items that loaded on more than one component were excluded from the analysis in order 
to have scales based on the items that capture the distinctness of the domains (2009, 744). 


However, as one can see, the two survey sets yielded different factor analysis results, and 

as such, reflect different survey items as markers of domain distinctness. 
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1. Survey Instrument Analysis Using Rotated Factor Analysis Results 

Observations of the final data sets, as represented in Tables 1 and 2, led to 
examination of the similarities between the rotated factor analysis results to determine 
which survey items were more effective in capturing self-efficacy in the intended 
domains. The survey items to keep, based on the rotated factor analysis results of each 
survey, would be: 

Teamwork 

• Help two team members resolve serious personal differences so they can 
work effectively with the team. 

• Help make sure your teammates deliver on the schedule that they agreed 
to support. 

• Talk with your team members for the purpose of assessing their interests 
and skills in order to ensure needed team capabilities are included. 

• Identify important personal relationships that must be considered to 
complete a project. 

• Get your team members at project start to make personal commitments to 
their responsibilities. 

• Give constructive criticism to a poorly performing team member that helps 
them improve their performance. 

• Facilitate a team to reach agreement when there had been conflicting 
views about what steps should be taken next. 

Systems Engineering 

• Generate alternative system ideas to meet a stakeholder need. 

• Elicit requirements that a future stakeholder has for a system, service, or 
construct using direct interaction with them. 

• Leverage features or elements of existing systems that can inform a new 
design. 

• Translate the system capabilities that you have co-developed with 
stakeholders into technical requirements. 

• Create a feasible design that meets the system requirements. 

• Pick a technology for a system considering that the technology readiness 
level is estimated, and uncertain. 

• Recognize when you can reduce the complexity of your completed system 
design. 
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• Prioritize the requirements that a future customer has for a system, service, 
or construct. 

• Design a system that fits within its desired total ownership cost. 

• Determine what evidence is required as data to demonstrate that a system 
meets its requirements. 

• Create a system architecture that has appropriate elements, attributes, and 
relationships to be able to be used for system development and decision 
making. 

• Formulate a methodology and approach for solving a systems engineering 
problem in an unfamiliar subject area. 

• Select the most appropriate initial design iteration to meet a given set of 
requirements. 

• Describe what resources will be necessary to conduct tests to demonstrate 
that a system meets its requirements. 

Critical Thinking 

• Find an inconsistency in an argument the first time you read a document. 

• Recognize any logical inconsistencies in a presentation by a speaker. 
Project Management 

• Recognize the important assumptions that need to be tested at the start of a 
new system development project. 

• Determine what measures must be taken in a risk mitigation strategy to 
assess if the likelihood or impact have been assuaged as development 
progresses. 

• Prioritize events in a risk mitigation strategy and ideate corrective action. 

This subset of survey items represents the skills that were observed to load with a 

.6 or higher in a single component, within both the rotated factor analysis tests. 

D. RESULTS: ANALYSIS AND IMPLICATIONS 

In general, there is some evidence to show that there was improvement in the 
systems engineering students as they progressed through the program, in their perceived 
level of self-efficacy to perform systems engineering skills. Although not all items 
displayed statistically significant differences between pre- and post-survey results, 
changes in mean scores (from pre- to post-) were generally reported to be higher, 
reflecting an overall enhanced perception of self-efficacy. 
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An interesting observation of the survey results is that the first cohort (RT19) 
generally marked their pre-scores to be much higher than those of the next cohort 
(RT19A). This trend leads us to examine possible reasons for the difference in perceived 
levels of self-efficacy between cohorts, as there are potentially a couple of reasons for 
this observation. For instance, in a discussion with a lead faculty member who 
interviewed the RT19 students about the scores, it was deemed that the students’ 
relatively high perception of their own ability could have been affected by the 
preconceived notion that, prior to any formal education or training, they had an ability to 
accomplish many of the competencies because of the context of their military 
experiences. This point is supported in the research review discussed earlier, in which we 
describe the process by which mastery experiences (in our case, that of the students’ 
having prior experience in performing SE related tasks, albeit through a military setting), 
can positively affect self-efficacy. It is meaningful to add to this discussion that there was 
a structured difference between the RT19 and RT19A course format, but the change in 
structural format was not deemed to have affected the RT19 cohort’s pre-survey results— 
and in fact the NFS research team believes it would have impacted the post-survey 
results. It should be noted that the RT19 pre-survey was deployed during the middle of 
the students’ first quarter. So, they had essentially already received a fair amount of 
educational experience in fundamental SE concepts, compared to RT19A. The RT19A 
cohort was given their survey at the beginning of their curriculum. The decision of the 
lead faculty to deploy the RT19A pre-survey at an earlier point in the students’ master’s 
program came as a result of an informal review process with students. It was deemed that 
adjusting the process and procedure when deploying to the RT19A cohort would control 
for the way students understood the questions. Also, the project that was used for the 
basis of learning (and by which the students would have been expected to base their 
experiences), was pedagogically designed as an authentic-learning project. 

To summarize, a general assessment of the high pre-scores of the RT19 cohort is 
that the students may have misjudged (i.e., overestimated) their skills and abilities in the 
pre-test, as a result of them feeling like they had experience in performing the tasks in a 
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military environment, but not so much in an educational environment. Again, we stress 
the role of mastery experiences in enhancing a student’s perceived level of self-efficacy. 

1. Teamwork 

Relevant research has suggested that teamwork provides students with an 
opportunity to evaluate the skills of their peers (Ponton et al. 2001, 250). When a student 
observes his/her peers with competencies he/she values, then the individual student can 
believe him/herself equally competent in performing the same skill (Ponton et al. 2001, 
250). As such, our hypothesis was that the survey results would indicate increased levels 
of self-efficacy in items relating to teamwork, as one would hope progressing through a 
master’s program at university would enhance opportunities and skills necessary for 
working in teams. In general, results indicate mixed findings as the results for the first 
cohort did not show significant differences in the average response rates towards many of 
the teamwork aspects, while the second cohort seems to have improved at significant 
levels. An area of teamwork that seems to have been enhanced is the students’ ability to 
determine an operating plan once all team members understand their interests and skills. 
This finding is indeed interesting as research has found teamwork in engineering has the 
advantage of solving complex problems (Ponton et al. 2001, 249). 

Among the first cohort’s results, the skill was observed to be significant at p<.01, 
with a percent increase in pre- and post-results of nine percent. The second cohort’s 
percent increase between pre- and post-average was 30 percent and was statistically 
significant at p<.001. An interesting observation one immediately notices when studying 
the results is the difference in the percent increase of pre- and post-means between 
cohorts. The second cohort seems to have indicated lower levels of self-efficacy in their 
pre-survey and so display a larger mean difference in post results. It is noteworthy to 
mention that the statistical method of using a t-test accounts for the difference in sample 
sizes, and so the difference in mean percent increase across teamwork items between the 
first and second cohort provides interesting context for further study. It should be noted 
that a possible explanation for this observation is that lead faculty incorporated course 
work that addressed teamwork into the curriculum of the second cohort (RT19A), but not 
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the first cohort. This was because, for the first cohort, it was assumed the students could 
effectively work in teams. This structural method may have impacted the results. 

2. Critical Thinking 

Data analyses of the first cohort’s results seem to indicate that the self-efficacy 
scores in critical thinking did not go up enough to exceed rejection criteria for traditional 
hypothesis testing, which would contend that there would be a significant difference 
between pre- and post-averages. All but two of the skills surveyed demonstrated higher 
averages in the post-survey results—meaning that even though the gains were not 
significantly different, there was at least a general sense of improvement among students. 

An analysis of the second cohort yields markedly different results, with most of 
the post mean differences relating to critical thinking self-efficacy skills statistically 
significant from pre-survey results at p<.001. 

The mixed results observed within the critical thinking items may be due to the 
social phenomenon that students are not aware of how well they can perform a certain 
task until they actually have to utilize the skill or knowledge set, in a work environment. 
Put in context, self-efficacy influences the level of accomplishment realized by people as 
they perform tasks (Bandura 1997, 3). 

In reviewing the critical thinking results, it should be noted that the structural 
format in teaching the first cohort involved referring the students to textbooks on the 
topic of engineering reasoning, whereas the second cohort was more formally taught 
from lectures (i.e., through parts of course work and seminars) on what critical thinking is 
and how it is used. The NPS research team feels that this change in structure may have 
impacted the survey results. 

3. Systems Engineering 

Our hypothesis was that the survey results would indicate increased levels of self- 
efficacy in items relating to systems engineering, as it is expected SE students will, at the 
least, feel efficient and capable in performing standard systems engineering tasks as they 
progress through a master’s program. 
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Contrary to results observed in the other systems engineering domains, the first 
cohort’s results showed significant differences in several areas. Two skills in particular 
showed significant differences between pre- and post-results at p<.01. These were the 
student’s ability to use projected warfighting capability gaps to estimate the future need 
for a new system and to describe what resources will be necessary to conduct tests to 
demonstrate that a system meets its requirements. One self-efficacy item in particular 
reflected statistically significant pre- and post-survey results at p<.001, which was the 
student’s ability to pick a technology for a system considering that the technology 
readiness level is estimated, and uncertain. 

The survey offered to the second cohort culminated in mean pre- and post¬ 
differences that went up enough to exceed rejection criteria for traditional hypothesis 
testing. There were significant differences in means observed in all self-efficacy items 
relating to systems engineering. These results may be due to the change in structural 
format of instruction delivery between that of the first and second cohort. The first cohort 
integrated their authentic project within each of the courses in the curriculum. The second 
cohort had an additional three consecutive project courses added to their curriculum in 
which the instructor reviewed all SE methods and processes in the context of the project. 
This facilitated near transfer of the learning of SE methods and processes to the authentic 
project context. So, in theory, key SE concepts were revisited a number of times. 
Therefore, the NFS team feels that this approach to teaching systems engineering 
concepts in context of authentic-learning projects seems to improve self-efficacy and is 
more effective in teaching. Also this finding supports our assumption that the systems 
engineering students would perceive higher levels of self-efficacy as they progressed 
through SE courses. As such, it can be assumed that the education provided to the 
students positively affected their ability to perform systems engineering tasks. 

4. Project Management 

Similar to results observed in the critical thinking self-efficacy items, most of the 
first cohort’s results did not go up enough to exceed traditional rejection criteria for 
hypothesis testing. An exception to this observation was in the skill of estimating 

quantitatively the likelihood and relative impact of events on a development project that 
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represent the most pressing risks, which had an increase of 11 percent, and the difference 
in mean scores was statistically significant at p<.05. 

Results of the second cohort’s survey items relating to project management, 
showed statistically significant differences in the mean scores at p<.001—exceeding 
rejection criteria for traditional hypothesis testing. This finding supports our assumption 
that systems engineering education provided to the students improved their ability to 
perform tasks related to project management skills. 

Similar to the analysis discussed on systems engineering, the project management 
results may be due to the change in structural format of instruction delivery between that 
of the first and second cohort. As in the case in the systems engineering domain, the 
second cohort was explicitly taught how SE skills relate to the project via learning 
activities. The NFS team feels that assisting them in near transfer of knowledge and 
ability to their project may have influenced the survey results to reflect an increase in 
self-efficacy. 

E. CHAPTER SUMMARY 

The aim of this research was to determine whether self-efficacy in systems 
engineering could be reliably measured via a survey instrument among SE students at the 
Naval Postgraduate School. 

Overall, the Systems Engineering Competency Survey instrument can be seen as 
an effective means to measure self-efficacy in systems engineering students. As discussed 
above, the survey was delivered in a pre- and post-mode to two consecutive SE cohorts 
and was designed to establish, with confidence, that the systems engineering program had 
a measurable influence in improving the knowledge, skills and abilities of students to 
perform standard systems engineering tasks. The results support the hypothesis that there 
was a general sense of improvement in the systems engineering students as they 
progressed through the program, in their perceived level of self-efficacy to perform 
domain specific systems engineering .skills. 
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Several format structures could have biased the data and this should be strongly 
considered when assessing the results of the research. Perhaps the strongest bias was the 
sheer length of the survey. Offered as a paper questionnaire, the survey initially had 65 
items. A bias that may have occurred due to the length of the first format was that 
students may have inadvertently or purposely circled answers on the scale, without 
considering the value, in an effort to quickly submit the lengthy survey. Despite this, we 
chose to keep all survey submissions as there is no way to prove that a student did or did 
not feel they could perform the skills at the level they indicated. 

To address the possible areas of bias, the survey has since been divided up into 
two surveys, so as to lessen the load and time needed to take the survey. We used the 
method of rotated factor analysis with the data obtained from the first and second SE 
Competency surveys to observe which survey items, if any, loaded on multiple 
components, and took those survey items out that loaded on more than one component. 
This method aimed to ensure that each item would measure self-efficacy in a specific 
domain of systems engineering, and aided in reducing the length of the survey. Also, the 
two surveys are now offered online, and are paired: systems engineering/critical thinking 
(‘systems thinking’) and project management/teaming. Despite this, given that the factor 
analysis reports of each cohort’s survey data set yielded different component loading 
results (as unique data sets were used as the basis to run the analysis), an important 
follow-on step would be to merge the results to create a survey instrument that can more 
effectively measure self-efficacy in systems engineering. 

Finally, the NPS research team assessed possible implications of changes in the 
educational content and delivery structure between the cohorts (i.e., to explicitly go over 
key SE concepts via projects, course work and seminars). In examining the mean 
differences in pre- and post-scores, it was observed that statistically significant 
differences (particularly within the systems engineering and project management 
domains), could have been due to this structural changes in teaching, as assisting students 
in near transfer of knowledge and ability, and including authentic-learning projects seems 
to have increased the perception of self-efficacy in students. 
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V. CONCLUSIONS, RECOMMENDATIONS, SUMMARY AND 
AREAS FOR FURTHER RESEARCH 

A. SUMMARY 

In this chapter, we will discuss conclusions, recommendations and further areas of 
research based on the results of the SE Competency Survey analyses. We will also 
provide a relevant discussion on current areas of research that tie into the NPS study and 
can support our findings and the subsequent recommendations for further areas of 
research. 

It can be observed from the survey research results previously discussed that self- 
efficacy can be used as an indicator of systems engineering competency level 
achievement. Self-efficacy, as it relates to research within systems engineering, was 
defined as an individual’s belief that he or she can be proficient in skills necessary to 
become a systems engineering professional. System’s engineering self-efficacy was also 
defined as the beliefs in one’s ability to learn systems engineering skills. Relevant 
research suggests that self-efficacy is influenced by one’s own sense of capability and by 
the education, training and support received by others. As such, self-efficacy can be used 
in systems engineering as a motivational predictor of how well an individual will perform 
tasks, and can be utilized and developed to leverage performance enhancing advantages 
(National Security Leadership Foundation n.d.). 

Self-efficacy and cognitive/affective processes were shown to be integral in 
indicating performance in SE related careers. For instance, relevant research suggested 
that self-efficacy is a latent, psychological trait, which serves as a regulator across a 
broad range of functioning, as the key determinant for cognitive and affective processes. 
The NPS team cited Prawat’s (1989) finding that cognitive and affective skills are 
determined to be vital in indicating performance in SE because having a broad range of 
KSAs enables knowledge acquisition and application. The NPS team theorized that the 
acquirement of domain specific knowledge has the advantage of allowing one to facilitate 
the performance of work. 
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A literature review of relevant studies on the topie of self-effieaey led the NPS 
team to believe that a gap exists between being able to assess SE KSAs and measure the 
eompeteney of students/entry level engineers as a way to prediet future eareer 
performanee. It was stated that researeh by Lueas et al. (2009) that foundational, as it is 
in the development of survey models to measure self-effieaey, does not answer whether 
or not self-effieaey requires a higher level of eognition. Researeh eonducted by NPS, 
whieh maps KSAs across SE competencies and categorizes the KSAs into SE domains, 
fills this gap. The NPS research team contends that the research aids in the identification 
of the characteristic experiences and knowledge students need to strengthen self-efficacy 
in technical and non-technical fields. 

B. CONCLUSIONS AND RECOMMENDATIONS 

A conclusion the research team came to after reviewing the results of the SE 
Competency Survey was that self-efficacy can be used in systems engineering to highlight 
areas of potential educational/training gaps in the development of systems engineers. It 
was observed that teamwork, in particular, is a domain entry level systems engineers can 
feel generally less competent in. Critical thinking and project management had mixed 
results of self-efficacy levels between the two cohorts, with one cohort indicating 
statistically significant increases in competencies and the other cohort, at best, indicating 
a general sense of improvement (considering the difference in pre- and post-mean 
scores), hollowing Bandura’s overarching recommendation that “a major goal of formal 
education should be to equip students with the intellectual tools, self-beliefs, and self- 
regulatory capabilities to educate themselves throughout their lifetime” (1993, 136) the 
research team would advise NPS faculty and program developers to focus on teamwork 
strategies. Eor instance, teamwork projects could be adjusted to include upper and lower 
classmen. Research has found that when teams are structured to include both upper and 
lower classmen, the lower classmen may develop a heightened sense of competency in 
regards to completing the education because they observe the upper classmen as models. 
This instills in them a “perseverant self-efficacy” (Ponton et al. 2001, 250). 
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Given the research findings, the NPS team further recommends survey methods to 
assess self-efficacy. As stated earlier, survey methods utilizing task specific measures in 
the form of a survey, might be superior to other approaches, such as ER, to self-efficacy 
assessment. Similarly, the NPS research team contends that alongside utilizing the 
COMPOSE model as a career development tool, the SE Competency Survey can be used 
to identify achievement of specific knowledge, skills and abilities as indicators of 
competency in performing systems engineering tasks. 

Research on the identification of professional development KSA has suggested 
that industry knowledge, alongside El, should be included as skill areas (Moore and Rudd 
2005, 69). Given the fact that the NPS model does not include industry knowledge or El 
as competency areas, a recommendation for future research would be to study the impact 
and feasibility of including the two constructs into the NPS model. 

C. AREAS FOR FURTHER RESEARCH 

The NPS’s research team contends that the top recommendation for an area for 
further research would be to have an engineering organization validate the SE 
Competency Survey instrument. The NPS research team contends that if possible within 
time, participant, and resource constraints, the SE Competency Survey can be used by 
other engineering disciplines as a model to implement the self-efficacy based method. It 
can also be tailored to meet an organization’s requirements with minimal effort/changes. 
In referencing Whitcomb, Khan and White (2013) and White’s (2013) research discussed 
in Chapter III, it should be noted that the KSAs identified in the COMPOSE model have 
been validated by a couple of notable engineering organizations. As such, the validation 
of the SE Competency Survey would be a natural extension of the self-efficacy research. 
Doing so would provide necessary insight for those in academia involved in the 
education, training and development of systems engineers, as well as for those involved 
in the hiring of systems engineers. To address the concern of Moore and Rudd (2005, 75) 
cited earlier in the thesis, that students in a study such as this are not likely to perceive 
themselves to be poor in a competency area they think is very important, another area of 
research could be to develop a subjective SE exam for students to take upon graduation. 
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Or, students could be asked to relate the perceived level of importance of a competency 
alongside how competent they feel in being able to perform it, for a correlation analysis. 
Similarly, the NPS research team recommends further research to include a correlation 
analysis between achievement within a particular SE domain and self-efficacy, as 
research indicates that a “high sense of efficacy in a particular domain may not 
necessarily translate into having [a] similar level of efficacy in another domain” (Loo and 
Choy 2013, 86). 

Finally, recalling the discussion on “high component loadings” and “domain 
specific KSAs” the SE Competency Survey can be further refined and the new survey 
sub-portions can be re-written to include KSAs that are representative of domains not 
currently addressed by the survey. For instance, the SE Competency Survey does not 
address El skill areas such as time management and balancing personal and professional 
lives, so perhaps this can be an area for further research. Keeping in mind that “in such 
difficult budget times, demands on faculty time continue to increase” (Moore and Rudd 
2005, 75), including El competencies may prove beneficial in providing information on 
professional skills development given the current economic climate. 
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